Crawl budget is the number of URLs a search engine like Google can and wants to crawl on your site within a given period. It is shaped by two things: how much crawling your server can handle, and how much Google wants to crawl your content.
For most small and medium websites, crawl budget is not something to lose sleep over. For large sites, sites that change constantly, or sites with lots of messy URLs, it can decide whether new pages get discovered in days or sit unnoticed for weeks.
What is crawl budget?
Google's crawl budget guide for large sites defines crawl budget as the set of URLs Googlebot can and wants to crawl. Crawling is the step where a search engine's bot fetches a page so it can later analyze it. Without crawling, a page cannot be indexed, and without indexing, it cannot appear in search.
Crawl budget applies per hostname. Google notes that a subdomain like shop.example.com has its own crawl budget, separate from www.example.com.
It is also not unlimited. Google has finite resources and the web is enormous, so Googlebot has to prioritize. Crawl budget is how that prioritization plays out for your site.
Why crawl budget matters
Google is direct about who needs to care. Its guide is aimed at roughly three kinds of sites:
- Large sites with more than about 1 million unique pages whose content changes moderately often, such as weekly.
- Medium or larger sites with more than about 10,000 unique pages whose content changes very rapidly, such as daily.
- Sites with a large share of URLs reported as "Discovered - currently not indexed" in Search Console.
If your site does not fit those descriptions, Google says keeping your sitemap up to date and checking your index coverage regularly is usually enough.
When crawl budget is a real constraint, the effects are concrete. New product pages take longer to appear. Price or stock changes lag in results. Old, removed pages linger. And if Googlebot spends its time on endless filter combinations, your most important pages can get crawled less often than they should.
This carries over to AI search too. Google's AI features documentation says pages must be indexed and eligible for a snippet to appear as supporting links in AI Overviews and AI Mode. Pages that never get crawled never get that chance.
How crawl budget works
Google describes crawl budget as the combination of two factors.
Crawl capacity limit
The crawl capacity limit, sometimes called host load, is the maximum crawling Google will do without overloading your server. It considers how many parallel connections Googlebot uses and how long it waits between fetches.
This limit moves. If your site responds quickly for a while, the limit rises. If responses slow down or your server returns 5xx errors or HTTP 429 (too many requests), the limit drops. Google also notes that its own resources are a constraint.
Crawl demand
Crawl demand is how much Google wants to crawl your site. According to Google, it is driven by:
- Perceived inventory. The URLs Google knows about. If many are duplicates or unimportant, Google wastes time on them.
- Popularity. URLs that are more popular on the internet tend to be crawled more often.
- Staleness. Google tries to recrawl pages often enough to pick up changes.
Site-wide events like a migration can temporarily increase demand as Google reprocesses URLs.
Putting them together
Even if your server could handle more, Google will crawl less when demand is low. And even when demand is high, it will not exceed what your server can comfortably handle. Your effective crawl budget sits where those two meet.
Crawl budget in practice
Imagine a generic online marketplace with 50,000 real product pages. Its category pages let shoppers filter by size, color, brand, price and rating, and every combination creates a new URL. Add sort options and session parameters, and the site can generate millions of crawlable addresses from those 50,000 products.
In Search Console, the team sees a growing pile of "Discovered - currently not indexed" URLs, and new products take weeks to show up. The Crawl Stats report shows Googlebot spending much of its time on filtered URLs.
The fix is to shrink the inventory. The team keeps a small set of valuable filtered pages that match real searches, uses canonical tags to consolidate near-duplicates, blocks infinite sort and session parameters in robots.txt, removes discontinued products with proper 404 or 410 responses, and keeps the XML sitemap limited to canonical URLs with accurate lastmod dates. Over time, Googlebot spends more of its visits on pages that matter.
How to optimize crawl budget
Google's guide groups its recommendations into managing your URL inventory and making crawling more efficient. Here they are as practical steps:
- Consolidate duplicate content. Use canonical tags or redirects so Google focuses on unique pages.
- Block URLs you never want crawled. Use robots.txt for things like infinite sort orders or internal search results, if you cannot consolidate them.
- Return proper status codes for removed pages. Use 404 or 410 for permanently removed content. Google notes that blocked URLs stay in the crawl queue longer than ones returning a 404.
- Fix soft 404s. Pages that say "not found" but return a 200 status keep getting crawled.
- Keep sitemaps current. Include the URLs you want crawled and use the lastmod tag for updated content.
- Avoid long redirect chains. Each hop costs a fetch.
- Make pages fast. Google's crawling myths and facts page confirms that faster loading and rendering let it crawl more. Good server performance and Core Web Vitals work tend to go hand in hand.
- Support HTTP caching. Returning 304 Not Modified for unchanged content helps Google skip reprocessing.
- Monitor the Crawl Stats report. Watch response times, error rates and what kinds of URLs Googlebot is fetching.
If Google is crawling too hard and straining your server, its guide to reducing crawl rate suggests temporarily returning 500, 503 or 429 responses. It warns not to keep this up for more than a day or two, because URLs that return these codes for multiple days may be dropped from the index.
Common crawl budget myths and mistakes
Google publishes a list of crawling myths and facts. A few are especially worth knowing:
- "Crawling is a ranking factor." False. Crawling is needed to be in results, but more crawling does not mean better rankings.
- "Small sites get crawled less." False. Google says sites with important, frequently changing content get crawled often regardless of size.
- "Using noindex saves crawl budget." Only partly. Google has to crawl a page to see the noindex rule, so it still uses a fetch.
- "robots.txt can free up budget for other pages." Google says temporarily blocking pages does not shift budget elsewhere unless you are already hitting your capacity limit.
- "Google respects crawl-delay." False. Google's crawlers do not process the non-standard crawl-delay rule.
- "Updating dates makes Google crawl more." False. Trivial edits or fake date changes add no value.
- "Compressed sitemaps increase crawl budget." False. Google still has to fetch the file.
- "4xx errors waste crawl budget." False, except 429. Google only receives a status code.
Crawl budget vs. indexing
These get mixed up constantly. Crawl budget is about whether Google fetches a page. Indexing is about whether Google decides to store and serve it. A page can be crawled and still not indexed, which Search Console reports as "Crawled - currently not indexed." That is usually a quality or duplication question, not a crawl budget one.
How Butter approaches crawl budget
At Butter, we start by asking whether crawl budget is actually your problem. For most sites we work with, it is not, and we will say so. The bigger wins are usually content quality, internal linking and clean indexing signals.
When crawl budget does matter, our monthly indexing strategy on the GEO plan looks at Search Console's indexing and crawl data to spot waste, like parameter sprawl or large clusters of discovered-but-not-indexed URLs. On the GEO+SEO plan, technical SEO fixes cover crawl issues directly, including redirect chains, broken links and soft 404s. The Butter Analytics dashboard connects to Search Console, so you can watch the trend instead of guessing. You can see how the plans compare on our pricing page.
Frequently asked questions
Do I need to worry about crawl budget?
Probably not, unless your site has around a million or more pages, more than about 10,000 pages that change daily, or many URLs stuck as "Discovered - currently not indexed." Google says most other sites only need an up-to-date sitemap and regular index monitoring.
How can I see my crawl activity?
The Crawl Stats report in Google Search Console shows how often Googlebot requests your pages, response times and errors. The Page Indexing report shows which URLs were discovered, crawled and indexed.
Can I ask Google to crawl my site more?
Google does not take requests to increase crawl rate. You can raise demand and capacity by improving server speed, removing low-value URLs and publishing content people actually seek out.
Does noindex save crawl budget?
Not directly. Google has to crawl a page to see the noindex rule, so the fetch still happens. Over time, removing pages from the index can reduce how often they are recrawled.