Crawl budget is the amount of pages Googlebot is ready and willing to crawl on your site within a given period. For most small and medium-sized sites this is not a problem; Google's own documentation states this plainly. On the other hand, for e-commerce, listing and news sites with hundreds of thousands of addresses, crawl budget can determine how quickly new pages are discovered. This article explains, without exaggeration, who this topic concerns, what wastes the budget, and which steps actually work.
What is crawl budget?
Googlebot does not crawl every address on your site every day. Its resources are limited and it distributes them across billions of sites. The amount of crawling allocated to a site is informally called crawl budget. The important point is this: budget is not a ranking factor. Being crawled more does not mean ranking higher; it only helps your pages get discovered and updated more quickly.
Two components: crawl capacity and demand
| Component | Meaning | Depends on |
|---|---|---|
| Crawl capacity limit | The number of simultaneous requests Googlebot can make without overloading your server | Server response speed, error rate (5xx), hosting resources |
| Crawl demand | How much Google wants to crawl your site | The site's size, popularity, how often content changes, and how "stale" it becomes |
The actual crawl volume is the smaller of these two. If your server slows down or frequently returns 5xx errors, Google lowers capacity. If your pages are low quality and unappealing, demand is low anyway. In other words, hosting performance determines capacity, and content quality and popularity determine demand.
Who does it concern?
According to Google's guide for large site owners, managing crawl budget is meaningful for these sites:
- Sites with roughly a million or more unique pages whose content changes at a medium frequency (such as weekly).
- Sites with ten thousand or more pages whose content changes very frequently (daily).
- Sites with many addresses in the "Discovered - currently not indexed" status in Search Console.
A corporate site, blog or small store with a few hundred or a few thousand pages does not fall into this group. On such sites, a new page being slow to enter the index usually stems not from crawl budget but from content quality, internal links or a technical obstacle. First eliminate the causes listed in our indexing issues guide.
Honest note: On a small site, worrying about "increasing my crawl budget" is often solving the wrong problem. First check whether the page is crawlable, indexable and genuinely valuable.
What wastes crawl budget?
The main sources of waste identified by Google are:
- Filter and sort combinations (faceted navigation): When color, size, price and sorting options in a category combine, thousands of parameterized addresses can be created for the same product list.
- Session IDs and tracking parameters: Structures that generate a different URL for every visitor.
- Duplicate content: The same page existing at several addresses. To consolidate, see the canonical tag.
- Soft 404 and 404 pages: Addresses that do not exist but keep being crawled. See 404 errors and SEO.
- Long redirect chains: Each step consumes a separate request. See the 301 and 302 difference.
- Endless spaces: Calendars and "next month" links that produce endless URLs.
- Low-quality and thin pages: They get crawled but never enter the index, consuming the budget.
- Slow server: As response time grows, Googlebot crawls fewer pages in the same time.
How is it measured?
- Crawl stats report: In Search Console, the "Crawl stats" report under Settings shows the daily request count, average response time, the distribution of response codes and file types. Start by looking at sudden changes in total requests and in the 5xx/404 ratios.
- Server logs: The only source that precisely shows which addresses are crawled and how often. From here you can see which parameterized addresses and which folders Googlebot visits most.
- Pages report: The number of "Discovered - currently not indexed" and "Crawled - currently not indexed" statuses, and their sample addresses.
- Sitemap comparison: Compare the number of addresses in the sitemap with the number of unique addresses crawled in the logs.
A hypothetical scenario: in a store with 20,000 products, if most of Googlebot's requests in the logs go to parameterized addresses such as
?color=...&size=...&sort=... and new products do not enter the index for days, the likely problem is that the budget is
being spent on filter combinations. (This is a hypothetical example; verify it on your own site with logs.)
Optimization steps
- Manage your URL inventory. Block areas you do not want crawled (internal search results, filter combinations with no value) using
robots.txt. For how to write the rules, see our robots.txt guide. - Consolidate duplicates. Gather addresses that lead to the same content into a single address using redirects or canonicals.
- Return 404 or 410 for permanently deleted content. Google does not forget a known address, but a 404/410 is a strong signal to reduce crawling of it.
- Clean up soft 404s. Empty pages or pages saying "not found" should return the correct status code.
- Keep the sitemap up to date. Include only addresses you want indexed that return 200, and match the
lastmodvalue to real updates. For details, see our XML sitemap guide. - Shorten redirect chains and point internal links directly to the final address.
- Improve page loading and server speed. As the server gets faster, the capacity limit may rise. Caching and hosting quality are decisive here.
- Highlight important pages with internal links. Pages close to the home page and to frequently crawled pages are discovered more easily. Our internal linking strategy article covers this topic.
- Monitor server health. Overload, 5xx and 429 responses cause Googlebot to slow down.
Example: managing a filtered category page
The most typical source of crawl budget problems is filtered navigation. If every combination of color, size, brand and sorting options in a shoe category produces a separate address, a list of one hundred products can turn into thousands of crawlable addresses. To decide, ask this question: Are people actually searching for this filter combination?
- Yes (e.g., "black running shoes"): Develop the page as a separate target page with unique title, description and content, support it with internal links, and keep it crawlable.
- No (e.g., "size 42 + price ascending"): Limit crawling with robots.txt, or prevent these addresses from being widely generated by on-page links; connect them to the main category with a canonical.
- Sort and session parameters should never be in the index; all of them should be consolidated into a single address.
Note: Google cannot see the canonical tag on addresses blocked by robots.txt. So do not rely on both blocking and canonical for the same addresses; make a conscious decision about which one you choose.
Common misconceptions
| Common belief | Reality |
|---|---|
| "Noindex saves crawl budget." | No. Googlebot still requests a noindexed page; it just does not index it. Robots.txt is used to reduce crawling, and noindex to remove from the index; know the difference between the two. |
| "If I am crawled more, I will rank better." | The amount of crawling is not a ranking signal. |
| "I can raise the crawl rate in Search Console." | Google has removed the old crawl rate limiter tool; the rate is determined automatically. |
| "Every site has a budget problem." | The topic does not apply to most sites; it is noticeable only on very large or very fast-changing sites. |
| "Submitting sitemaps often increases crawling." | A sitemap helps discovery, but does not force crawling. What matters is that the sitemap is accurate and up to date. |
All these steps are part of your technical audit; for a general order of checks, use our technical SEO checklist. On JavaScript-generated sites, crawling and rendering take extra time, so also take a look at our JavaScript SEO basics article. Google's official guide is on its page about managing crawl budget for large sites.
Frequently asked questions
How do I find out my crawl budget?
There is no single fixed number. The Crawl stats report in Search Console shows the daily request count; server logs give the detail address by address.
Does crawl budget matter for my small site?
Usually not. For sites up to a few thousand pages, the focus should be on content quality, internal linking and basic technical health.
Does a page I block with robots.txt leave the index?
No. Robots.txt prevents crawling, but if that address receives external links it can remain in the index. To remove it from the index, use noindex, and for that the page must be crawlable.
Does speeding up my server increase the budget?
When your response time drops, Google can crawl more pages in the same time, so the capacity limit gains potential to rise; but it is not a guaranteed result.
Conclusion
Crawl budget is a real topic that large sites must manage, and a concept that small sites mostly should not chase. Evaluate it according to your site's size: if it is large, monitor the crawl stats and logs and reduce unnecessary addresses; if it is small, put your energy into content and internal links. In both cases, a fast server, a clean URL structure and an up-to-date sitemap make Googlebot's job easier. To see what you can build on top of a technical foundation to grow visibility, you can browse our organic traffic growth methods article.