Claude is the AI assistant and family of large language models built by Anthropic. People use it through the Claude apps on web, desktop and mobile, and developers use it through Anthropic's API, where it can search the web and cite the sources it used.
This page explains what Claude is, how its web search and citations work according to Anthropic's documentation, which crawlers Anthropic runs and how to manage them in robots.txt, and what all of this means for brand visibility.
What is Claude?
Anthropic describes Claude in its developer documentation as an AI platform suited to language, reasoning, analysis and coding tasks. Claude comes in several models that trade off capability, speed and cost; Anthropic publishes the current lineup on its models overview page.
Consumers reach Claude at claude.ai and through apps for macOS, Windows, iOS and Android, according to Anthropic's download page. Businesses also build Claude into their own products through the Claude API and through cloud platforms such as Amazon Bedrock and Google Cloud's Vertex AI.
For marketers, Claude belongs in the same group as ChatGPT, Gemini, Perplexity and Copilot: an assistant people ask for recommendations, explanations and comparisons, and one that can pull live information from the web when answering.
Why Claude matters
When Claude answers with web results, it shows citations to the pages it used. That makes it one more place where your brand can be mentioned, recommended and linked, or left out. Because Claude is also built into other companies' products through the API, its reach extends beyond the Claude apps themselves.
Claude also matters for technical SEO because Anthropic runs its own crawlers. Decisions you make in robots.txt affect whether Claude can index your site for search, fetch your pages when a user asks and use your content for training.
Keep expectations grounded. Anthropic does not publish how Claude ranks or selects sources, and it does not offer a webmaster console for Claude like Google Search Console. Brands can make their content accessible and credible, but they cannot control whether Claude cites them.
How Claude works
Web search and citations
Anthropic launched web search in Claude in March 2025 and says it is available on all plans. According to Anthropic's help center article on web search, responses that use web search include citations, source links and relevant quotes so users can verify them. Users can switch search on or off per chat in some versions of the app, while in others Claude decides on its own when a search would help. On Team and Enterprise plans, an owner must enable web search for the organization first.
On the developer side, Anthropic's web search tool documentation says Claude decides when to search based on the request, can run several searches in one response, and always returns citations that include the source URL, page title and a short snippet of cited text. Developers can limit results to, or block, specific domains.
Reading specific pages
Claude can also fetch and read a specific URL. Anthropic's web fetch tool documentation notes that it handles HTML, text and PDFs but does not currently support pages that are rendered dynamically with JavaScript, and that robots.txt restrictions apply.
Anthropic's crawlers
Anthropic's help article on how it crawls the web and how to block it lists three user agents, each with a different job:
- ClaudeBot collects public web content that may be used to train Anthropic's models. Blocking it signals that your future content should be left out of training data.
- Claude-User fetches pages when a Claude user asks a question that needs them. Blocking it stops Claude from retrieving your pages for those requests, which may reduce your visibility in user-directed answers.
- Claude-SearchBot crawls to improve Claude's search results. Blocking it keeps your content out of that index, which may reduce your visibility in search results.
Anthropic says its bots honor standard robots.txt directives, respect anti-circumvention measures such as CAPTCHAs, and support the non-standard Crawl-delay directive. Rules must be set for each subdomain. Anthropic notes that blocking by IP address may not work reliably, and it publishes a list of its crawler IP addresses for verification. Search Engine Journal reported in February 2026 that Anthropic had expanded the documentation from one listed crawler to these three, so blocking the training bot no longer has to mean blocking search.
Claude in practice: an example
Take a hypothetical software company that blocked ClaudeBot in robots.txt to keep its content out of AI training. It later notices that Claude rarely cites its pages, even for branded questions, and wants to make sure nothing else is in the way.
Reviewing its setup, the team finds that a broad bot-protection rule at its CDN is also challenging Claude-SearchBot and Claude-User. It keeps the ClaudeBot opt-out, explicitly allows the other two user agents in robots.txt, and fixes the CDN rule. It also checks that its pricing and feature pages load their key content in plain HTML, since some fetchers do not run JavaScript.
The company then runs a monthly set of prompts in Claude with web search on and tracks whether its pages begin appearing as cited sources. There is no guarantee they will, but the technical barriers are gone.
How to optimize for Claude
- Decide your crawler policy bot by bot. Choose separately whether to allow ClaudeBot (training), Claude-SearchBot (search indexing) and Claude-User (user-requested fetches), based on your goals. Blocking all three limits your visibility in Claude.
- Audit your robots.txt and CDN rules. Check that you are not blocking Anthropic's user agents by accident through broad AI crawler rules, firewall settings or bot protection. Remember each subdomain needs its own rules.
- Keep key content in HTML. Put important facts, prices and definitions in the server-rendered page, not behind scripts, tabs or logins.
- Write clear, quotable passages. Claude's citations point to specific snippets of text. Short, self-contained statements of fact are easier to cite accurately.
- Build credibility off your site. Claude, like other assistants, draws on third-party sources. Reviews, press coverage and accurate listings shape how it describes you.
- Test in Claude regularly. Run real customer questions in Claude with web search enabled, note whether you are mentioned and cited, and repeat on a schedule.
Common mistakes and misconceptions
"Blocking ClaudeBot removes us from Claude." Not necessarily. Anthropic documents ClaudeBot as its training crawler. Claude-SearchBot and Claude-User are separate user agents with separate rules, so you can opt out of training while staying available for search and user requests.
"Claude only knows what it was trained on." With web search, Claude can pull current information from the web and cite it. Its training data still shapes answers when it does not search.
"Claude uses Google results, so Google SEO is all we need." Anthropic's documentation does not say which search provider powers Claude's web search, so do not plan around assumptions. Solid SEO fundamentals help across AI tools, but test Claude directly.
"There is a way to submit our site to Claude." Anthropic does not document any site submission process for Claude. Accessibility for its crawlers, clear content and third-party credibility are the levers you have.
Claude vs. ChatGPT
Both are general-purpose AI assistants that can search the web and cite sources, and both come from companies that run separate crawlers for different purposes. OpenAI's crawler documentation lists user agents including GPTBot, OAI-SearchBot and ChatGPT-User, while Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User. For brands, the takeaway is the same: manage each bot deliberately and test visibility on each platform separately, since they retrieve and cite differently. See ChatGPT Search for the OpenAI side.
How Butter approaches Claude
We treat Claude as one of several AI platforms a brand needs to be visible on, and we start with fundamentals: crawler access, indexable content and accurate information across the web.
Our GEO plans start at $399/month and include monthly AI prompt testing, indexing strategy, schema markup guidance, knowledge graph submissions and on-page recommendations. Prompt testing shows how AI assistants describe and cite a brand, and indexing strategy covers decisions like which AI crawlers to allow.
We are careful not to overstate what anyone can control. We can remove technical barriers and strengthen your content and reputation, but how Claude chooses its sources is up to Anthropic.
Frequently asked questions
Does Claude search the internet?
Yes. Anthropic added web search to Claude in 2025, and responses that use it include citations to the sources. Depending on the app and plan, users can toggle it per chat or Claude decides when a search would help.
How do I block Claude from crawling my website?
Add rules for Anthropic's user agents to your robots.txt, for example a "User-agent: ClaudeBot" line followed by "Disallow: /" to opt out of training. Use separate Claude-SearchBot and Claude-User rules to control search indexing and user-requested fetches, and repeat the rules on each subdomain.
Does blocking ClaudeBot stop Claude from citing my site?
According to Anthropic's documentation, ClaudeBot is the training crawler, while Claude-SearchBot and Claude-User handle search and user requests. Blocking only ClaudeBot does not block the other two.
Who makes Claude?
Claude is made by Anthropic, an AI company. It is available through Anthropic's own apps and API, as well as through cloud platforms such as Amazon Bedrock and Google Cloud's Vertex AI.
Sources and further reading
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Claude Help Center
- Enable and use web search, Claude Help Center
- Web search tool, Claude Platform Docs
- Web fetch tool, Claude Platform Docs
- Claude web search now available globally on all plans, Anthropic
- Anthropic's Claude Bots Make Robots.txt Decisions More Granular, Search Engine Journal