Crawl Budget: Who Actually Has a Problem, and the Index Bloat Behind It

Crawl budget is how much crawling Google is willing and able to do on your site. It has two halves, both named in Google’s own large site owner’s guide: the crawl capacity limit, which is how hard Google can push your server without degrading it, and crawl demand, which is how much Google actually wants your pages. Capacity is a technical ceiling. Demand is an editorial judgement, and it is the one people ignore.

The first thing to establish is whether this applies to you at all, because for most sites it does not, and the time spent on it is time stolen from something that would have worked.

Google’s own threshold

Google opens that guide by telling most readers to leave. In its words, you do not need to read it if your site does not have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published. It then names who it is for:

Large sites, upwards of a million pages, with content changing about weekly.
Medium or larger sites, upwards of ten thousand pages, with content changing daily.
Sites with a large proportion of URLs reported as Discovered, currently not indexed.

That third bullet is the one that matters in practice, because it is a symptom rather than a size. A twelve-thousand-URL ecommerce site with eight thousand URLs stuck in Discovered has a crawl problem. A forty-page consultancy site publishing twice a month does not, no matter what a crawler tool’s warning panel says.

The calculation that settles it in five minutes

This is arithmetic, not an algorithm secret, and I am surprised how rarely it gets published. Open the Crawl Stats report in Search Console, take the average daily crawl requests, and divide your indexable URL count by it. That gives you the number of days it takes Google to get round your whole site once.

Indexable URLs	Avg. daily crawl requests	Days for one full pass	Reading
400	250	1.6	No crawl problem exists. Stop here.
8,000	2,000	4	Healthy. Changes are picked up within the week.
40,000	2,000	20	Watch it. A price change takes three weeks to register.
250,000	3,000	83	Real problem. Most of the site is effectively frozen.
250,000	3,000, of which 70 percent hit junk URLs	278 on the pages that matter	This is the actual shape of most cases.

The last row is the whole point. Raw crawl volume is rarely the constraint. Where the crawl goes is. I have audited sites where Googlebot spent the clear majority of its requests on filter combinations, session parameters, paginated archives and soft 404s, and the client’s conclusion was that they needed more crawl budget. They did not. They needed fewer URLs.

Index bloat is the disease, crawl budget is the symptom

Index bloat is what you have when a site exposes far more URLs than it has distinct things to say. Faceted navigation is the classic engine: four filters with six values each, combinable, and a fifty-product category page has quietly generated thousands of near-identical URLs. Add sort orders, pagination and a session parameter and the number stops being countable.

Every one of those URLs competes for the same crawl, dilutes internal link equity, and gives Google more opportunities to conclude that this site is mostly padding. My position is that crawl budget work which does not start by reducing the URL space is decoration. Compressing responses and trimming redirect chains helps at the margin. Deleting sixty thousand junk URLs changes the shape of the site.

Google’s best practices, and what each one is actually for

Google publishes nine. Here they are with what each one buys you, in my order of leverage rather than theirs.

Practice	What it buys	Leverage
Consolidate duplicate content	Removes whole classes of URL from contention	Highest
Block unwanted URLs with robots.txt, not noindex	Google must crawl a page to see a noindex, so noindex does not save crawl at all	Highest, and the most commonly got wrong
Return 404 or 410 for permanently removed pages	Tells Google to stop coming back	High
Eliminate soft 404s	Stops empty pages being treated as real content worth recrawling	High
Keep sitemaps current with accurate lastmod	Directs demand at what actually changed	Moderate
Avoid long redirect chains	Each hop is a request spent on nothing	Moderate
Improve page load speed	Raises the capacity limit, so Google can fetch more per unit of server time	Moderate
Use HTTP caching and 304 responses	Unchanged pages cost almost nothing to revisit	Moderate, and badly underused
Debug crawl issues in Crawl Stats	Tells you where the requests are actually going	Do this first, even though Google lists it last

The second row deserves its own sentence, because it is the mistake I correct most often. Noindex does not save crawl budget. Google has to fetch the page to read the directive. If your goal is to keep Googlebot away from a URL class, robots.txt is the tool; noindex is for keeping something out of the index that you still want crawled and followed. Confusing the two produces sites that are both bloated and slow to update.

What crawl budget is not

Being crawled more is not a ranking signal. A page that Googlebot visits daily does not outrank one it visits monthly by virtue of the visit. What crawl frequency gives you is latency: how quickly a change you make becomes a change Google knows about. On a news site or a large catalogue with moving prices, that latency is worth real money. On a stable brochure site it is worth almost nothing, and optimising it is a way of looking busy.

Crawl demand, though, does move with quality, and that is the part worth internalising. Google describes demand as varying with a site’s size, update frequency, page quality and relevance compared with other sites. Quality is in that list. Which means a crawl budget problem on a large site is frequently a quality problem wearing a technical costume, and technical fixes will only take it so far.

The order I work in

Run the calculation above. If a full pass takes under a week, close the tab and go and do something useful.
Segment Crawl Stats by response and by purpose. You are looking for the share of requests hitting URLs you would never want ranked.
Count the URL space against the content space. How many distinct things does this site have to say, and how many URLs does it publish? The ratio is the finding.
Kill the generator, not the output. Blocking ten thousand facet URLs while the facets keep producing more is treating a symptom.
Serve 410 for what is genuinely gone. Do not redirect junk into healthy pages to make the report look tidy, which is how a contained problem spreads.
Fix soft 404s and redirect chains. Cheap, mechanical, worth doing once the big cuts are made.
Then look at speed and caching. Real, but last, because it raises the ceiling on a crawl you have now pointed somewhere sensible.
Re-run the calculation after four weeks. Crawl patterns take time to settle, and changing more than one thing at once destroys attribution.

Step four is where most engagements go wrong. Faceted navigation, calendar archives, internal search results and tag pages are generators. Until the generator is configured, you are shovelling.

Two related pieces. If you are working through a site systematically rather than chasing a specific symptom, the crawl and architecture pass sits inside my wider SEO audit checklist. And if the URL sprawl came from pages generated per query variant rather than from faceted navigation, that is a spam policy issue rather than a crawl one, and it belongs with the scaled content problem I describe in how to rank in AI Overviews.

Finally, the diagnostic caution I apply to all of this: a crawl anomaly that appears at the same moment as a traffic drop is not necessarily its cause. Establish which came first before you commit a quarter to it. That sequencing is the whole of my method in Google penalty recovery.

Royking Niba is an SEO and GEO consultant specialising in penalty and spam-update recovery, with more than 8 million organic visits recovered for clients. Google’s crawl budget guidance quoted here was checked against its published documentation on 22 September 2026.