Crawling is the process search engines use to discover and download pages on the web. Automated programs, called crawlers or spiders, follow links from page to page and fetch what they find so it can be processed later.
Crawling is the first step in search. If a page isn't crawled, it can't be indexed, and if it isn't indexed, it can't rank in Google or be retrieved by most AI search tools. Understanding how it works helps you remove the obstacles that keep good pages from being found.
What is crawling?
In search, crawling means a bot requests a URL, downloads its content and looks for links to other URLs. Google's How Search works guide describes it as the first of three stages, followed by indexing and serving results. Google's main crawler is Googlebot.
There is no central registry of every web page. Google says it has to keep searching for new and updated pages, finding them mainly by following links from pages it already knows, and from sitemaps site owners submit. That is why links and sitemaps matter so much for discovery.
Why crawling matters for search and AI answers
Crawling decides what a search engine even knows about. A brilliant page with no links pointing to it, or one blocked by a misconfigured server, may simply never be seen. New pages, updated prices and refreshed content only show up in search once they are recrawled.
AI search depends on crawling too. OpenAI documents a dedicated bot, OAI-SearchBot, that surfaces websites in ChatGPT search, and says sites that block it won't be shown in ChatGPT search answers, though they may still appear as navigational links. Perplexity documents PerplexityBot for surfacing sites in its results. Google says pages in its AI features must be indexed and eligible to appear in Search, which starts with crawling. Being crawlable is a prerequisite for showing up in generative engine optimization (GEO) work.
How crawling works
URL discovery
Crawlers build a queue of URLs to visit. Sources include links on already-crawled pages, XML sitemaps and previously known URLs. Google can reliably follow links only when they use a standard a element with an href attribute, per its link best practices. Links built only with click events may be missed.
Checking permission
Before fetching a URL, a well-behaved crawler checks the site's robots.txt file. If the path is disallowed for that crawler, it skips the URL. Pages behind a login also can't be crawled.
Fetching and rendering
The crawler downloads the HTML and the resources it needs. According to Google's JavaScript SEO guide, pages that return a 200 status are queued for rendering, where a headless version of Chromium runs the JavaScript. Google then parses the rendered HTML for more links. Rendering can be delayed until resources are available.
Deciding how much and how often
Google uses an algorithm to choose which sites to crawl, how often and how many pages to fetch. It also slows down when a server struggles. Google's crawl budget guide frames this as a combination of crawl capacity (what your server can handle) and crawl demand (how much Google wants to crawl, based on factors like popularity and staleness).
Crawling in practice
Consider a hypothetical publisher that launches a new resource section with 200 guides. The guides are linked only from a JavaScript-powered filter widget that doesn't output real links, and they are missing from the sitemap. Weeks later, most still show as "Discovered" or not found at all in Search Console.
The fix is ordinary: add the guides to the sitemap, link to them from relevant category pages with standard HTML links, and confirm the server responds quickly. Once crawlers can reach the pages through normal paths, discovery usually follows.
How to make your site easy to crawl
- Link to every important page. Google recommends each important page be linked from at least one other page. Strong internal linking is the most dependable discovery signal you control.
- Use crawlable links. Stick to a elements with real href URLs.
- Keep a clean sitemap. List canonical, indexable URLs and keep it updated automatically.
- Check robots.txt. Make sure you aren't blocking important sections, CSS or JavaScript.
- Keep the server healthy. Slow responses and 5xx errors cause Google to crawl less.
- Limit duplicate URLs. Faceted filters, session IDs and endless calendar pages can create huge numbers of low-value URLs that soak up crawler attention.
- Monitor it. The Crawl Stats report in Search Console shows requests, response times and errors. Google notes sites under about a thousand pages usually don't need it, but it is useful for larger sites.
Common misconceptions about crawling
- "Crawled means indexed." Not necessarily. Google may crawl a page and decide not to index it.
- "Blocking a page in robots.txt removes it from Google." Blocking stops crawling, not indexing. Use a noindex tag to keep a page out of results.
- "Small sites need to worry about crawl budget." Google says its crawl budget guidance is aimed at very large or fast-changing sites. Most small sites get crawled fine if pages are linked and the server is healthy.
- "You can pay or request Google to crawl more." You can request indexing of individual URLs, but you can't ask Google to raise your overall crawl rate.
Crawling vs. indexing
Crawling is fetching the page. Indexing is analyzing it and storing it in the search engine's database so it can be shown in results. A page has to be crawled before it can be indexed, but crawling doesn't guarantee indexing. Quality, duplication and directives such as noindex all affect the second step.
How Butter approaches crawling
At Butter, crawlability is part of the indexing strategy we work through each month. We check that search engines and the AI crawlers that matter to you can reach your key pages, and we look for the usual blockers: stray robots.txt rules, broken links, orphaned pages and server errors. On our GEO+SEO plan, fixing crawl issues is part of the technical SEO work included. We can't control how often Google visits, but we can make sure nothing on your side is getting in the way.
Frequently asked questions
How often does Google crawl a website?
It varies. Google decides crawl frequency algorithmically, based on factors like how often a site changes, how popular it is and how well the server responds. Frequently updated, well-linked sites tend to be crawled more often.
How can I get a new page crawled faster?
Link to it from relevant pages on your site, make sure it is in your XML sitemap, and request indexing with the URL Inspection tool in Search Console. None of these guarantee immediate crawling, but they help Google find it.
Do AI tools crawl websites the same way Google does?
The basics are similar: bots fetch pages and follow rules in robots.txt. But each AI company runs its own crawlers with different names and purposes, so check each one's documentation and decide which to allow.