A robots.txt file is a small plain-text file at the root of your website that tells crawlers which parts of the site they may and may not request. It is one of the oldest tools in SEO, and it is still one of the easiest to get wrong.
Used well, robots.txt keeps bots out of low-value areas and protects your server. Used carelessly, one line can block your whole site from Google. This guide explains what the file does, what it does not do, and how to set it up safely.
What is robots.txt?
robots.txt is a text file that follows the Robots Exclusion Protocol, a convention that was formally published as an internet standard, RFC 9309, in September 2022. It lives at the top level of a host, for example https://www.example.com/robots.txt, and contains groups of rules addressed to specific crawlers.
According to Google Search Central, the file's main job is to manage crawler traffic so bots don't overload your server. It controls crawling, meaning whether a bot fetches a URL. It does not directly control indexing, meaning whether a URL can appear in search results. That distinction is the source of most robots.txt mistakes.
Why robots.txt matters for search and AI
For traditional search, robots.txt helps you steer crawlers away from URLs that waste their time, such as internal search results, filtered product listings or cart pages. On large sites this can support a healthier crawl budget, because Googlebot spends more of its visits on pages you actually want found.
For AI search, robots.txt has become the main way site owners signal what AI companies may do with their content. Different AI crawlers use different user-agent tokens. OpenAI, for example, documents separate bots for ChatGPT search (OAI-SearchBot) and for model training (GPTBot), and each can be allowed or blocked independently. Google offers a token called Google-Extended for Gemini training and grounding, and Google states it does not affect inclusion or ranking in Google Search, per its crawler documentation.
How robots.txt works
Location and scope
The file must sit in the root directory of the host. A copy in a subfolder is ignored. Rules apply only to the exact protocol, host and port the file is served from, so blog.example.com and www.example.com each need their own file, as Google's robots.txt specification explains.
The four supported fields
Google supports four fields: user-agent (which crawler the group applies to), disallow (paths not to crawl), allow (paths that may be crawled, usually as an exception), and sitemap (the full URL of your XML sitemap). A simple group looks like this: "User-agent: *" on one line, then "Disallow: /cart/" on the next. Google ignores other fields, including crawl-delay.
How conflicts are resolved
A crawler follows only the most specific user-agent group that matches it. If a file has a group for Googlebot and a group for all bots (*), Googlebot uses only its own group. Within a group, the longest matching path wins, and when an allow and a disallow rule are equally specific, Google applies the less restrictive allow rule. Paths are case-sensitive, and the wildcards * (any characters) and $ (end of URL) are supported.
Limits and error handling
Per Google's robots.txt documentation, Google reads up to 500 KiB of the file and ignores anything after that. It generally caches robots.txt for up to 24 hours. If the file returns a 404, Google assumes there are no restrictions. If it returns a server error, Google pauses crawling at first and then falls back to the last cached copy, so a broken robots.txt can quietly slow down crawling of your whole site.
robots.txt in practice
Imagine a mid-sized online store. Its faceted navigation creates thousands of near-duplicate URLs, such as every combination of color, size and price filter. Its server logs show bots spending much of their time on these filter pages instead of new products.
A sensible robots.txt might disallow the internal search path and the filter parameters, allow everything else, and list the sitemap URL at the bottom. The store would keep product and category pages fully crawlable, and it would test the rules before publishing to make sure no important path was caught by a wildcard.
How to set up robots.txt correctly
- Start permissive. Block only what you have a clear reason to block. Most small sites need very few rules.
- Never block CSS or JavaScript your pages need. Google renders pages, and blocking key resources can make them harder to understand.
- Add your sitemap. Include a "Sitemap:" line with the absolute URL. It can point to any host.
- Decide on AI crawlers deliberately. Choose which search and training bots to allow based on your goals, rather than copying someone else's file.
- Test before you ship. Use the robots.txt report in Google Search Console and spot-check key URLs with the URL Inspection tool.
- Review after migrations. Staging sites often ship with a "Disallow: /" rule that accidentally goes live.
Common robots.txt mistakes
- Using robots.txt to hide pages from Google. Google says robots.txt is "not a mechanism for keeping a web page out of Google." A blocked URL can still be indexed, often without a description, if other pages link to it. Use a noindex tag or password protection instead.
- Blocking a page that has a noindex tag. If Google can't crawl the page, it never sees the noindex rule.
- Treating it as security. RFC 9309 states the rules are not a form of access authorization. Bad bots can simply ignore them, and the file itself is public.
- Adding a noindex line to robots.txt. Google does not support it.
- Relying on crawl-delay for Google. Googlebot ignores it.
How Butter approaches robots.txt
At Butter, robots.txt is one of the first files we check, because a single bad line can undo every other SEO and GEO effort. On our GEO+SEO plan, robots.txt problems fall under the technical fixes we handle, alongside broken links and other crawl issues. We look at whether Google and the AI search crawlers you care about can reach your important pages, and we make sure blocks are intentional rather than leftovers from a redesign.
We also talk through the AI crawler question with each client. Allowing search bots while restricting training bots is a business decision, and we explain the trade-offs rather than pushing one answer. If you want a second set of eyes, our SEO service covers this as part of regular technical reviews.
Frequently asked questions
Does every website need a robots.txt file?
No. If there is no robots.txt file, Google assumes it can crawl everything, which is fine for many small sites. A file becomes useful when you need to keep crawlers out of specific areas or want to list your sitemap.
Can robots.txt remove a page from Google?
Not reliably. Blocking crawling does not stop a URL from being indexed if other pages link to it. To remove a page from results, allow crawling and use a noindex tag, or protect the page with a password.
Should I block AI crawlers in robots.txt?
It depends on your goals. Blocking a search-focused bot such as OAI-SearchBot can keep your pages out of that tool's answers, while blocking training bots such as GPTBot only affects model training. Many businesses that want AI visibility allow search bots and make a separate choice about training.
How quickly does Google notice robots.txt changes?
Google generally caches robots.txt for up to 24 hours, so changes usually take effect within about a day. You can request a recrawl of the file through the robots.txt report in Search Console if you need it picked up sooner.