Googlebot is the name of Google's main web crawler, the automated program that visits pages across the web and downloads them so Google Search can index them. If you see "Googlebot" in your server logs, Google is fetching your content.
Knowing how Googlebot behaves helps you make sure it can reach, render and understand your most important pages. It also helps you spot fake bots pretending to be Google.
What is Googlebot?
Googlebot is the generic name for two crawlers, per Google Search Central: Googlebot Smartphone, which simulates a mobile user, and Googlebot Desktop, which simulates a desktop user. Because Google primarily indexes the mobile version of most sites, the majority of requests come from the smartphone crawler.
Both share the same robots.txt token, "Googlebot," so you can't allow one and block the other with robots.txt. Google also runs related crawlers with their own tokens, such as Googlebot-Image, Googlebot-Video and Googlebot-News, listed in its common crawlers documentation.
Why Googlebot matters
Googlebot is the gateway to Google Search. Pages it can't fetch can't be properly indexed, and pages that aren't indexed can't rank. Google notes that blocking Googlebot also affects Discover, Google Images, Google Video and Google News.
It matters for AI answers too. Google says a page must be indexed and eligible to show in Search with a snippet to appear in its AI features, such as AI Overviews, and it advises allowing crawling in robots.txt. So content Googlebot can't reach is unlikely to appear there. Separately, Google offers a token called Google-Extended to control whether content is used for Gemini model training and grounding, and Google says that token has no effect on Search inclusion or ranking.
How Googlebot works
Discovery and the crawl queue
Googlebot finds URLs by following links on pages it already knows and by reading submitted sitemaps, as described in Google's How Search works guide. An algorithm decides which sites to crawl, how often and how many pages to fetch.
Respecting robots.txt
Before fetching a URL, Googlebot checks the site's robots.txt file. Google classifies Googlebot as a common crawler, which always respects robots.txt rules for automatic crawls.
Fetching with limits
Per Google's Googlebot documentation (updated February 2026), Googlebot crawls the first 2MB of a supported file type, measured uncompressed, and the first 64MB of a PDF. Resources such as CSS and JavaScript are fetched separately, each under the same 2MB limit. For most sites, Google says Googlebot shouldn't visit more than once every few seconds on average. It supports gzip, deflate and Brotli compression, and HTTP caching with ETag and Last-Modified headers, per its crawler overview.
Rendering
Googlebot renders pages with a recent version of Chrome so it can see content loaded by JavaScript. Rendering may happen after the initial crawl, once Google's resources allow.
Googlebot in practice
Imagine an online retailer notices a spike in "Googlebot" requests hammering its search pages. Before blocking anything, the team verifies the traffic. A reverse DNS lookup shows many of the IPs don't resolve to Google domains, so the traffic is a scraper using a fake user agent. The team blocks those IPs at the firewall and leaves the real Googlebot alone.
If the traffic had been genuine, blocking Googlebot outright would have been a costly mistake. The better fix would be to find the root cause, often faceted navigation creating endless URLs, and address it with robots.txt rules or cleaner URL handling.
How to work well with Googlebot
- Serve the same core content on mobile. With most crawling done by Googlebot Smartphone, your mobile pages should carry your full content, links and structured data.
- Don't block essential resources. Let Googlebot fetch the CSS and JavaScript needed to render your pages.
- Keep pages lean. Put important content well within the first 2MB of HTML.
- Use crawlable links and a clean sitemap. Standard HTML links and an accurate XML sitemap help Googlebot find pages.
- Verify before you block. Use Google's verification steps: run a reverse DNS lookup on the IP, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup to confirm it matches. Google also publishes IP range files.
- Slow it down safely if needed. If Googlebot overloads your server, Google's crawl rate guide suggests temporarily returning 500, 503 or 429 status codes, but not for more than a day or two, because prolonged errors can lead to URLs being dropped.
Common misconceptions about Googlebot
- "Every visitor named Googlebot is Google." User agents are easy to fake. Always verify.
- "Blocking Googlebot removes my pages from Google." Blocking crawling doesn't stop a URL from appearing in results. Use a noindex tag for that.
- "Crawl-delay controls Googlebot." Google doesn't support crawl-delay in robots.txt.
- "Blocking Google-Extended hides me from Search." Google says it doesn't affect Search inclusion or ranking.
How Butter approaches Googlebot
At Butter, we treat Googlebot access as a baseline check in the indexing strategy we review each month. We look at whether Googlebot can reach and render your key pages, review Search Console for crawl errors, and flag robots.txt or server issues that could be holding pages back. On the GEO+SEO plan, fixing those crawl problems is part of the included technical SEO work. We can't influence how Google chooses to crawl, but we can remove the barriers on your side.
Frequently asked questions
How do I know if Googlebot has visited my site?
Check your server logs for the Googlebot user agent and verify the IPs, or open the Crawl Stats report in Google Search Console. The URL Inspection tool also shows when Google last crawled a specific page.
Can I block Googlebot Desktop but allow Googlebot Smartphone?
Not with robots.txt. Both use the same "Googlebot" token, so a rule for one applies to both. Google tells them apart only by the full user-agent string.
Does Googlebot run JavaScript?
Yes. Googlebot renders pages using a recent version of Chrome. Rendering can be delayed, so critical content and links in the initial HTML are found most reliably.