Crawl Budget: What It Is and When to Optimise It

Crawl budget is the set of URLs that Googlebot can and wants to crawl on your site within a given time. It comes down to two things: the crawl capacity limit (how much your server can ‘take’) and crawl demand (how keen Google is to visit your pages). Only large and fast-changing sites really need to worry about it – for a typical company website or blog, an up-to-date sitemap and a regular look at the Search Console reports are enough.

In the world of SEO you’ll come across plenty of technical terms that sound complicated at first but turn out to be pretty straightforward once you break them down. One of them is crawl budget. Below I explain how it works according to Google’s current documentation (as of October 2026), who it really affects, what wastes it, how to check it and which popular ‘tips’ are myths.

What is crawl budget?

The internet is practically infinite, so Google can’t regularly visit every publicly available URL. The time and resources it can devote to a single site are limited – and that allocation is what’s called crawl budget. Google defines it as ‘the set of URLs that Googlebot can and wants to crawl’.

A quick note on terms: crawling and indexing are two different stages. Crawling is the bot fetching a page; indexing is the later decision about whether the page goes into the index. Not every crawled page gets indexed – after fetching it, Google evaluates it, consolidates it with duplicates and only then decides. So crawl budget doesn’t determine what ends up in the index, only what Google gets round to looking at in the first place.

An important detail: the budget is calculated per hostname. To Google, www.example.com and shop.example.com are separate sites with separate budgets.

History of the term

The term ‘crawl budget’ was doing the rounds in the SEO community long before Google referred to it. It was used as a loose name for the mechanisms that limit what bots do. The official explanation appeared on 16 January 2017 on Google’s blog (the post ‘What Crawl Budget Means for Googlebot’). Back then Google described two components: crawl rate limit and crawl demand.

Today the topic has its own guide, ‘Optimize your crawl budget’. In December 2025 Google moved it, together with its other crawling documents, from Search Central to a separate site about its crawling infrastructure: developers.google.com/crawling/docs/crawl-budget. That’s a sign these rules apply not just to Search, but to all of Google’s crawlers.

Crawl rate limit – what is it?

Crawl rate limit is the old name for the crawl capacity limit, i.e. the upper bound on how intensively Googlebot can fetch pages from your server without slowing it down. Today Google calls it the crawl capacity limit (or hostload). The limit isn’t fixed – it goes up when the server responds quickly and reliably, and goes down when 5xx errors, 429 codes or slow responses appear. You can’t set it manually (more on that in the section on reducing crawling below).

The components of crawl budget

Google distinguishes two main components that affect crawl budget:

• Crawl capacity limit (formerly crawl rate limit, also called hostload): determines how long your server can keep connections with Google’s crawlers open – both the number of parallel connections and their duration count. Every site starts with the same cautious limit. If the server responds steadily and response times (including time to first byte) don’t get worse, the limit goes up. When the server slows down or returns 5xx errors or a 429 code, the limit goes down and Google crawls less. This limit is shared by all of Google’s crawlers, so a big appetite on the part of one of them (e.g. AdsBot) can shrink the pool for the rest.
• Crawl demand: this is a crawler’s ‘desire’ to visit your URLs – unlike the capacity limit, each Google crawler has its own crawl demand. For Googlebot it depends on the size of the site, how often it’s updated, and the quality and relevance of its pages. Google lists three factors you can influence: perceived inventory (all the URLs Google knows about – if many of them are duplicates or junk, the bot wastes time on them; this is where you have the most control), popularity (popular URLs are visited more often) and staleness (Google comes back to pick up changes). Demand also rises temporarily during site-wide events, such as a move to a new domain.

To sum up: even if the server could handle more, Google will visit the site less often when demand is low. And the other way round – high demand won’t help if the server can’t keep up.

	Crawl capacity limit	Crawl demand
The question it answers	How much can Google crawl without overloading the server?	How much does Google want to crawl?
What it depends on	Speed and stability of server responses, 5xx errors, 429 codes, Google’s own resources	The number of known URLs (and their quality), popularity, staleness, events such as a site move
Shared between crawlers?	Yes – one pool for all of Google’s crawlers	No – each crawler has its own
How to improve it	A faster, more stable server, fewer errors	Fewer duplicates and junk URLs, better content quality

Who does crawl budget affect?

The good news is that most websites – local business sites, industry blogs or shops with a limited range of products – don’t need to worry about crawl budget. Google says outright that if your pages tend to be crawled on the day they’re published, you don’t need to read its crawl budget guide. The Search Console help page for the Crawl Stats report also gives a threshold: if your site has fewer than 1,000 pages, you most likely don’t need this level of detail.

Google’s guide is aimed mainly at:

• large sites (over a million unique pages) whose content changes roughly once a week,
• medium and larger sites (over 10,000 unique pages) whose content changes daily,
• sites where a large share of URLs have the status ‘Discovered – currently not indexed’ in Search Console.

Google points out that these are rough figures, not hard thresholds. In my experience, the problem shows up earlier wherever the number of ‘technical’ URLs snowballs – in shops with filters, classified ad sites or portals with event calendars. That’s when internal structure, redirects, the robots.txt file and the quality of server responses start to matter.

Estimating crawl budget

There’s no single metric that shows ‘crawl budget’ as a number. You can, however, estimate it by analysing data from Google Search Console and your server logs. In the Crawl Stats report you’ll see, among other things, the total number of Googlebot requests, the amount of data downloaded and the average server response time. From that you can tell how often Google comes back to the site and whether anything is changing. Server logs give you an even more accurate picture – by comparing Googlebot’s visits with the list of URLs you actually want in Google, you can check how much of the budget goes to important pages and how much to parameters, duplicates and errors.

What eats up crawl budget?

Google’s general rule is: in general, any URL that Googlebot crawls counts towards the budget. The things that waste it most often are:

• URL parameters and faceted navigation: filters, sort orders and their combinations (e.g. ?colour=green&size=s&sort=price) can generate a practically infinite number of URLs. The bot doesn’t know whether they’re valuable until it fetches them, so it fetches loads of them – and there’s less time left for new, important pages. Calendars with a separate URL for every date work the same way.
• Duplicate and low-value content: duplicates (versions with session parameters, different sort orders of the same list, infinite-scroll pages that repeat content) make Google fetch many URLs with the same content. This is exactly the ‘perceived inventory’ over which you have the most control.
• Soft 404s: pages that look like an error (‘no products’, ‘not found’) but return a 200 code. Google says outright that such pages will keep being crawled and waste the budget.
• Redirect chains: every URL in a chain is a separate request – the Crawl Stats report counts them separately. Google points out that long redirect chains have a negative effect on crawling.
• Server errors and slow responses: a large number of 5xx errors, connection timeouts or 429 responses signals to Google that the server can’t cope – the capacity limit drops and the bot crawls less. 404 and 410 errors are a different story: according to Google, pages returning 4xx codes (except 429) don’t waste crawl budget, because the bot gets only the code, with no content.
• Heavy resources and rendering: CSS and JavaScript files (including XHR requests) and alternate URLs such as AMP or hreflang also count towards the budget. Rendering time counts just like fetch time, so a page that’s hard to render ‘costs’ more.
• Very large files: this isn’t so much wasting budget as a trap worth knowing about. Since February 2026 Google has documented that when crawling for Search, Googlebot fetches the first 2 MB of a supported file (e.g. HTML) and the first 64 MB of a PDF; each CSS and JS file has the same 2 MB limit separately, counted on the uncompressed data. Anything beyond that is ignored. By default Google’s crawlers fetch the first 15 MB of a file, but individual crawlers may have their own limits.

The consequences of poor crawl budget management

Crawl budget isn’t a ranking factor, but crawling a page is its entry point into the search engine – if Googlebot doesn’t visit a page, it has no chance of being indexed and showing up in the results. On a large site, using the budget inefficiently means new products or articles wait a long time to be discovered, price and availability changes reach Google late, and some important URLs are left with the status ‘Discovered – currently not indexed’. Google also warns that if the bot spends too much time on URLs it shouldn’t be crawling, it may not get to the rest of the site or may not increase the budget.

Where to check crawl budget and indexing

The most important tool is Google Search Console. You’ll find the Crawl Stats report in the property’s Settings (Settings → Crawl stats). It shows the history of Googlebot’s visits: the total number of requests, the size of the downloaded data, the average response time, host status (problems with robots.txt, DNS and server connectivity) and a breakdown of requests by response code, file type, purpose (discovery or refresh) and Googlebot type. The report is available only for top-level properties – a Domain property or a URL-prefix property at root level.

The second report is Page indexing (formerly ‘Coverage’ or ‘Crawl errors’). There you’ll find URLs with 404 and 5xx errors, soft 404s, canonical issues, robots.txt blocks and URLs with the status ‘Discovered – currently not indexed’. This is where you’ll spot the places that waste budget fastest – especially soft 404s and server errors. It’s also worth looking at the Sitemaps section to check that your submitted sitemaps are read without errors.

Server logs complement this by showing Googlebot’s actual visits, URL by URL. For smaller sites GSC data is usually enough; for large shops, log analysis is one of the first steps I take as part of an SEO audit.

Technical tips for optimising crawl budget

For crawl budget to work in your favour, it’s worth looking after the technical side of your site. Here’s a list of recommendations based on Google’s guide:

• Correct robots.txt configuration: block URLs Google shouldn’t crawl at all – e.g. filter combinations, different sort orders of the same list or internal search results. Don’t, however, block normal category pagination, because that’s how the bot reaches products and articles. Google also makes it clear that robots.txt isn’t for temporarily ‘shifting’ budget to other pages – freed-up budget is only reallocated if the site is already hitting its capacity limit. Make sure important pages aren’t blocked by accident.
• An up-to-date sitemap: Google reads your sitemap regularly, so include all the URLs you want crawled. For updated content Google recommends the <lastmod> tag – provided it reflects a real change to the content, not every page refresh.
• Server performance: optimise server response time and resources (caching, compression, leaner code, a CDN). A fast and stable server tells Google it can open more connections. Time to first byte is particularly worth keeping an eye on. Also support the 304 Not Modified code: if a page hasn’t changed, Google will use the cached version, saving bandwidth and server resources. Google’s crawlers support the ETag and Last-Modified headers – Google recommends setting both, and when both are present it uses ETag. Optionally, you can add max-age in Cache-Control to hint how long the content will stay unchanged.
• Avoid duplicate content: consolidate duplicate pages (redirects, correct canonical tags, tidy URL parameters). Make sure product content or articles don’t exist in many identical versions. Remember that a canonical tag doesn’t block crawling – Google still has to fetch the duplicate to see the tag. If duplicates can’t be consolidated and you don’t want them crawled, Google recommends a robots.txt block; according to the current guide, it also significantly reduces the chance that such URLs will be processed by other Google systems.
• Manage page removals: if you remove a page permanently, return a 404 or 410 code (410 means ‘permanently removed’). Google doesn’t forget a URL it knows, but a 404 is a clear signal to it not to crawl it again. URLs blocked in robots.txt stay in the queue much longer and go back to being crawled once the block is lifted. You can read more about the differences between these codes in my article on 404, 410 and other 4xx errors. With faceted navigation, return a 404 for filter combinations with no results instead of redirecting to a shared error page.
• Get rid of soft 404s: a ‘not found’ page must return a real 404/410 code, not 200. You’ll find a list of soft 404s in the Page indexing report.
• Optimise redirects: limit the number and length of redirect chains. Every redirect is a separate request, so redirect straight to the target URL, update internal links to the final URLs and don’t create loops. I’ve described the differences between 301, 302, 307 and 308 in a separate post on 3xx redirect status codes, and how to bulk-check status codes for a list of URLs in my guide to a Python script.
• Monitor the stats: keep an eye on the Crawl Stats report and host status. React to sudden drops in the number of requests, rising response times or a growing number of 5xx errors – they may signal a server failure or a configuration error. If the URL Inspection tool shows you the message ‘Hostload exceeded’, Google explicitly recommends considering a more powerful server.

Google names only two ways to increase crawl budget: adding server resources (when the server is the bottleneck) and improving content quality – for the search engine, things like popularity, value to users and uniqueness of content count. There’s no setting or form you can use to ‘ask’ for a bigger budget.

How to reduce crawling when Googlebot overloads your server

The crawl rate limiter tool in Search Console was retired on 8 January 2024. First, check your logs to see where the traffic is coming from – according to Google, the most common causes of a sudden spike are faceted navigation, calendars with URLs for every date and Dynamic Search Ads. If you need an emergency response, Google recommends (in its Reduce Google’s crawl rate document) returning 500, 503 or 429 codes to the crawlers (with 503 and 429 you can add a Retry-After header) – but only for a few hours, 1–2 days at most. Any longer and URLs risk dropping out of the index. If that’s not possible, there’s a special form for reporting excessive crawling.

Crawl budget myths

Google has a separate page, ‘Myths and facts about crawling’, covering a dozen or so myths. The key points:

• ‘noindex saves crawl budget’ – partly true. To see a noindex, Google has to crawl the page, so noindex isn’t a tool for controlling crawling – in its guide Google explicitly advises against using it that way. In the longer run, pages removed from the index may be crawled less often, so noindex can indirectly free up a little budget. If a page shouldn’t be in the index, use noindex and don’t worry about the budget. If it shouldn’t be crawled, use robots.txt.
• ‘404 errors waste crawl budget’ – false. 4xx codes (except 429) don’t waste the budget. Soft 404s and 5xx errors are the problem.
• ‘Small sites are crawled less often than large ones’ – false. If a site has important, frequently changing content, Google visits it often regardless of its size.
• ‘A compressed sitemap increases crawl budget’ – false. A zipped sitemap still has to be fetched from the server; the saving is negligible.
• ‘Frequent small changes and refreshing the date will attract the crawler’ – false. Artificially ‘refreshing’ pages adds no extra value.
• ‘Site speed and errors affect crawl budget’ – true. A fast server means more pages crawled in the same time; lots of 5xx errors and timeouts mean fewer. I’ve covered what these errors are and how to fix them in my post on 5xx server errors.
• ‘Pages closer to the homepage are more important’ – partly true. URLs linked from the homepage may be crawled more often, but that doesn’t mean they’ll rank higher.
• ‘Google doesn’t like parameters in URLs’ – false. Google copes fine with parameters. The problem isn’t parameters as such, but an uncontrolled number of combinations.
• ‘Changing a URL (e.g. adding a version number) forces a recrawl’ – partly true. It’s usually unnecessary and wastes resources. Only change the URL when the content has changed significantly.

FAQ – common questions about crawl budget

Is crawl budget a ranking factor?

No. Crawling a page more often won’t translate directly into higher rankings. Google says that crawling is necessary for a page to appear in the results at all, but it isn’t a ranking signal in itself. Crawling and indexing are just the start of the fight for visibility.

Do embedded content and alternate URLs count towards crawl budget?

Yes. In general, any URL Googlebot crawls counts towards the budget. That includes alternate URLs (AMP, hreflang) and embedded content – CSS and JavaScript files, including XHR requests.

Can I control crawling with the ‘crawl-delay’ directive?

No. Google’s crawlers ignore the non-standard crawl-delay rule in robots.txt (other search engines may respect it). If Googlebot really is overloading your server, temporarily use 503 or 429 codes, as I described above.

Do nofollow links affect crawl budget?

Partly. Every crawled URL uses up budget, and a link marked as nofollow may still be crawled if another page – on your site or anywhere on the web – links to it without nofollow. You’ll find the details on Google’s page Myths and facts about crawling.

Can you ask Google for a bigger crawl budget?

No. There’s no form or setting that will increase your budget. Google names only two routes: adding server resources if the server is what limits crawling (in that case the URL Inspection tool shows a ‘Hostload exceeded’ message), and improving content quality. The only form that exists works the other way round – for reporting that Googlebot is crawling your site too intensively.

Does a small company website need to optimise crawl budget?

Usually not. With a few dozen or a few hundred pages, Google can easily crawl the whole site. If such pages aren’t in the index, the cause is usually content quality, duplication or a technical error (e.g. noindex, a robots.txt block), not a lack of budget.

Running a large shop or site where new pages wait ages for Googlebot and the ‘Discovered – currently not indexed’ report just keeps growing? Get in touch about SEO consulting – we’ll check whether crawl budget is really the problem, or something else entirely.