Crawlability problems are the reasons Googlebot doesn’t get a usable copy of a page: it can’t find the URL, can’t fetch it, gives up on the way, or fetches it and is told not to keep it. On a working site most crawlability problems are self-inflicted. A Disallow written for staging and never removed, a noindex on a template nobody remembers setting, a redirect added on top of three older ones. Google changed nothing; the site’s own rules did. Fourteen of them below, with where each one shows in Search Console and what fixes it, though not every one has a tidy report to point at, and a couple only show up in a crawl you run yourself.
How to find crawlability issues before fixing them
Three places, and I’d open them in this order. Page indexing, under Indexing in Search Console, lists every URL Google knows about by status, and the crawl-side statuses have their own names: Blocked by robots.txt, Server error (5xx), Blocked due to access forbidden (403), Redirect error, Not found (404), Soft 404, Discovered – currently not indexed, Excluded by ‘noindex’ tag. Crawl stats, under Settings, is the crawl itself: requests per day, response codes, the share of requests spent discovering new URLs against refreshing known ones, and a host status that goes red when robots.txt, DNS or the server fails. The third is a crawl of the site with JavaScript off, in Screaming Frog or similar. It’s the least convenient of the three and the only one that catches the problems the reports never list, because the URLs were never reached in the first place. The fourteen below run roughly in the order Googlebot meets them, and the statuses map onto them like this:
| Status in Page indexing | What happened | Problem |
|---|---|---|
| Blocked by robots.txt | Googlebot read the rule and didn’t fetch the page | 1 |
| Server error (5xx) | Fetched; the server failed | 2, 3 |
| Blocked due to access forbidden (403) / unauthorized request (401) | Fetched; the server refused | 4 |
| Redirect error | A chain or loop past ten hops | 5 |
| Not found (404) | Fetched; nothing there. A crawlability problem only when the site links to it | 6 |
| Discovered – currently not indexed | Known, not fetched: the crawler has more URLs than it will spend on | 9, 10 |
| Soft 404 | Fetched with a 200 and read as empty | 12 |
| Excluded by ‘noindex’ tag | Fetched and told not to keep it | 13 |
| No status at all | Never reached: orphans, links that aren’t links | 7, 8 |
The 14 crawlability problems and how to fix each one
1. robots.txt blocks the page, or the files it needs
Three versions of the same mistake. A Disallow left from staging. A rule written for one folder that matches another by prefix, because Google matches on the path start and Disallow: /app also blocks /apple-watch/. And a block on /assets/ or /wp-content/, so the HTML loads and the CSS and JavaScript that build the page don’t. The first two show as Blocked by robots.txt. The third shows nowhere; the page renders wrong and nothing in Page indexing says so, and URL Inspection’s page resources list is the only place the blocked files appear.
Google reads four fields in the file, user-agent, allow, disallow, sitemap, and when two rules match one URL the longer one wins, so a short Disallow and a longer Allow can both apply and the Allow takes it. Since 2023 the tester is the robots.txt report in Search Console; the separate tool is gone, and the report shows the versions Google fetched and when, which settles the “but we changed that last week” argument. One thing the file never does is remove a page from the index: a blocked URL can stay indexed from links alone, with no content, because robots.txt controls the crawl and says nothing about indexing.
2. Server errors and host overload
A URL that returns 5xx is retried and, if it keeps failing, dropped; as crawlability problems go, the single URL is the small part. The larger effect is on the site: a rising share of 5xx tells Googlebot the host can’t take the load, and the crawl rate for every URL goes down, which Google’s crawl budget documentation describes as capacity adjusting to server health. Page indexing lists the URLs under Server error (5xx); the host status in Crawl stats goes red when the rate is high enough to matter. Usual sources: a database that fails under parallel requests, a deploy that returned 503 for an hour, a rate limiter counting Googlebot as abuse. Fix the source, allowlist Googlebot by the ranges Google publishes in googlebot.json so the limiter stops matching it, and watch the response-code chart for a week before deciding it worked.
3. Slow responses that lower the crawl rate
Same mechanism, no error. When response times rise, Googlebot sends fewer requests, and the way to see it is two lines in Crawl stats, average response time against total requests: when the first climbs and the second falls, the limit is being applied. Google publishes no threshold and I don’t have one either; the seconds quoted in guides are their authors’ guesses. Where the slow pages are depends on the site. On a store it’s the filtered category views, because each one runs the product query again; on a blog it’s the archive templates. Since Google’s July 2026 rewrite of the crawl budget documentation, the capacity is also shared per hostname across all of Google’s crawlers, so on a site running Shopping campaigns the ads and product fetchers draw from the same pool as Googlebot, and a slow server throttles all of them at once. Cache what can be cached, return 304 Not Modified for pages that haven’t changed, and stop the crawler reaching the slow views that shouldn’t be crawled at all, which is problem 10.
4. 403, 401 and the firewall
Googlebot asks and the server refuses. A WAF rule matching the user agent, a CDN bot setting, a geo-block on the United States, where Googlebot mostly crawls from, HTTP authentication meant for staging and shipped to production. Page indexing has these under Blocked due to access forbidden (403) and unauthorized request (401), and Crawl stats shows the day they started, which is usually the day of a deploy or a CDN plan change. Allowlist by published IP ranges rather than by user-agent string, which anyone can fake, and check the 403 rate by user agent in the logs afterwards. Since 2025 some CDNs block AI crawlers by default on new sites. Googlebot isn’t in that group, but the search crawlers from the assistants are, and a page they can’t fetch can’t be cited by them, so the same review covers both.
5. Redirect chains and loops
Googlebot follows up to ten hops and then records a Redirect error, and the ten is beside the point, because every hop before it is a separate request against the crawl budget. The chain I find on nearly every migration is four hops before the redirect anyone actually wrote: http to https, to www, to the trailing slash, to the new URL. Each rule was added at a different time and none of them knows about the others. Gary Illyes showed timing ranges at Search Central Live in Barcelona in October 2026, as attendees reported them, since no Google document carries the figures: a site move settles in one to three months in the typical case and six months to over a year in the slowest, and the chains left behind are the usual reason a site lands in the slow half. Collapse each chain to one 301, then update the internal links so the chain isn’t entered at all. A crawler’s redirect-chain report lists both halves, and the second half is the one that gets skipped.
6. Broken internal links and 404s
A 404 on its own is a URL that doesn’t exist, and Google treats that as normal. It becomes a crawlability problem when the site links to it, because then every visit to the linking page spends a crawl request on nothing and ends the crawler’s path. After a migration there are usually hundreds, in old posts, in widgets, in the footer. Crawl the site, export the links whose target returns 404, and restore the page, redirect it to its replacement, or remove the link; Not found (404) in Page indexing lists the URLs, but only the crawl lists the links that point at them.
7. Orphan pages
The quietest crawlability problem. A page nothing links to is discovered only from the sitemap, if it’s there, and a page the sitemap doesn’t have and nothing links to isn’t discovered at all. Orphans come from CMS pages published outside the navigation, old campaign landing pages, products that fell out of every category. No Search Console report shows them, which is the limitation worth knowing: Search Console describes URLs Google reached, and an orphan is defined by not being reached. The only way I know to find them is a diff, a crawl of the site against the sitemap and against the URLs in the Performance report. The URLs that get impressions and have no inbound internal link are the live orphans, pages still earning something from a place nobody can reach. Each needs an internal link from a page Googlebot visits often, which for most sites means a category page or the homepage.
8. Links that aren’t links
Googlebot discovers URLs from <a href> and nothing else on the first pass. A product card that opens on a click handler, a “load more” button that fetches the next twenty items without changing the URL, a menu built from <div>s with JavaScript routing, infinite scroll with no paginated URLs underneath. The page may be rendered later and some links recovered, but rendering is queued behind crawling, and links found only after rendering are found late. A crawl with JavaScript off shows what Googlebot sees first; if the product pages aren’t in it, they aren’t linked, whatever the browser shows. The fix belongs to the build: a real href on every link the crawler needs, paginated URLs under infinite scroll, and that rule in the acceptance criteria before launch, as the web development and SEO guide sets out.
9. Click depth and deep pagination
Googlebot reaches pages roughly in order of how many links they sit from the homepage and how often the linking pages are recrawled. A blog post on /blog/page/45/ is linked from one archive page that is itself linked from page 44, and gets crawled when that chain does, which is rarely; a product reachable only through a subcategory’s fourth page is the same shape. Google gives no maximum depth, and I wouldn’t trust a guide that does. What the “by purpose” chart in Crawl stats shows is that discovery crawls are a small share of the total, so a new page far from a frequently refreshed one waits. Flattening everything is the wrong response. Link new pages from pages that get refreshed, the homepage, the category page, the latest posts, and keep the archive pagination crawlable without depending on it.
10. Faceted navigation, parameters and duplicate URLs
If the site is under a few thousand pages, skip this one; it’s a duplicate problem there, not a crawl problem. Above that it’s the crawlability problem that eats the budget. Filters that each add a parameter combine into more URLs than the catalogue has products, colour, size, brand, sort order, price, in any order, each a distinct URL with a slightly different list. Add internal search results, session IDs and a calendar with an endless “next month” link, and the crawler has an infinite site. This is where Discovered – currently not indexed piles up: Google has more URLs from the site than it’s willing to fetch. Google’s own line is that crawl budget becomes a concern past roughly a million pages, or ten thousand that change daily, or any site with a large share of that status.
The controls, in order of how much each saves. Don’t link to the combinations in the first place; a filter that changes the page without a crawlable link generates no URL. Disallow the parameter patterns in robots.txt for the ones that must exist; this is the one job robots.txt does well. Canonical to the clean URL for the ones users share. The URL Parameters tool that older guides send you to was removed in 2022, and a few of those guides are still on the first page for this query.
11. A sitemap that lies
Nothing errors here, which is why it lasts. The sitemap is a list of URLs you’d like crawled, and Google trusts it to the degree it’s been accurate. One with 404s, redirecting URLs, blocked URLs, or a lastmod that moves on every deploy teaches Google to discount it, and then the one thing it’s for, getting new URLs fetched, stops working. Google ignores priority and changefreq and uses lastmod only while it’s consistently true. Generate it from the CMS on publish with only canonical, indexable, 200 URLs in it. The Sitemaps report shows what was read and how many URLs were discovered from it; a crawl of the sitemap URLs shows which don’t return 200.
12. Soft 404s
A page returns 200 and Google reads it as empty. A category with zero products, a search page with no results, a “not found” template served with a success code, a JavaScript page whose content didn’t render. They go into Soft 404 in Page indexing and get crawled less over time, and the ones that are real pages, a category temporarily out of stock, take the real pages down with them. Return 404 or 410 when there’s nothing there, a 200 with content when there is; a category with no stock keeps its description and its links and says so.
13. A noindex left on pages that should rank
I flag this on nearly every audit, and it’s rarely deliberate. A staging setting that shipped. A plugin that noindexes tag archives and took the category pages with them. A template flag set in 2022 by someone who has since left. Googlebot fetched the page, read the tag and was told not to keep it: the crawl succeeded and the index was refused, which puts this one on the border of what counts as a crawlability problem, and nothing errors, so a site that noindexed the wrong template can sit there for months. Page indexing lists them under Excluded by ‘noindex’ tag, and the export is worth reading by template rather than by URL, because the mistake is almost always a template.
The combination that hurts most is a noindex on a page also blocked in robots.txt. Googlebot can’t crawl it, so it never reads the tag, and an already-indexed page stays in the index with the one instruction that would remove it unread. One signal per URL: block to save crawl, noindex to clear something already indexed, never both on the same page.
14. nofollow on internal links
Still on the lists, with an outdated reason. Since 2020 Google treats rel="nofollow" as a hint rather than a directive, for crawling as well as ranking, so a nofollowed internal link doesn’t reliably stop the crawl, and it does reliably tell Google you don’t vouch for the target, which on your own link is an odd thing to say. Sites pick it up from a plugin setting or a widget template. Find them in a crawl, take them off internal links, and use robots.txt or noindex for pages that shouldn’t be crawled or indexed, since those do what nofollow is being asked to do.
What isn’t a crawlability problem
Several items on the usual lists of crawlability issues describe pages Googlebot fetched without difficulty. Crawled – currently not indexed is Google declining a page after reading it, usually a quality or duplication judgement, and no crawl fix moves it. Duplicate content and canonical conflicts are decisions about which fetched page to keep. Core Web Vitals and mobile layout are ranking inputs on an indexed page. Each has its place in the technical SEO audit guide, in the index and mobile steps, and sending them to a crawl checklist is how a team spends a week on robots.txt for a page Google read and didn’t want.
The split matters for one practical reason. The crawlability problems above are almost all fixed by developers and DevOps, in server configuration, templates and the CMS; the index-side problems are mostly fixed by whoever owns the content. A list that mixes them gets handed to the wrong team.
Crawlability problems and the AI crawlers
The same fourteen apply to the agents behind ChatGPT, Claude and Perplexity, and three of them bite harder there. The search crawlers from the assistants (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch HTML and, in every test published so far, don’t render it, so a link that only exists after JavaScript runs, problem 8, isn’t found late for them; it isn’t found. Since July 2025 Cloudflare has blocked AI crawlers by default on new sites, so problem 4 can be a CDN setting the robots file knows nothing about, and the readout is the 403 rate by user agent in the logs. And robots.txt rules written per agent, problem 1, often date from 2023, when a GPTBot Disallow was copied to every agent that followed. The agents and what each one does are in the ecommerce SEO audit; none of this is a separate AI workstream; the same crawlability issues get read for one more set of user agents.
Crawlability problems FAQ
What’s the difference between crawlability and indexability?
Crawlability is whether Googlebot can discover and fetch the page; indexability is whether, having fetched it, Google adds it to the index. Most of the problems above sit on the first side, and a few, noindex among them, on the border, where a successful crawl is followed by a refused index.
How do I check whether Google can crawl my site?
URL Inspection on a handful of templates: last crawl, response, whether robots.txt allowed it, which resources were blocked. Then Crawl stats for the trend and the host status, then Page indexing for the statuses above. A crawl with the Googlebot user agent and JavaScript off catches what those don’t, mainly links that aren’t links.
Does crawl budget matter for a small site?
Rarely as budget, though crawlability still does. Google’s guidance is that sites below a few thousand pages are crawled in full, and the crawl-rate limit only becomes a constraint for very large or very fast-changing sites. The individual problems above apply at any size, because a blocked, orphaned or looping page isn’t reached whatever the budget.