robots.txt for AI crawlers: what to allow

AI companies don't send one bot. Most send several, each with its own job: one collects pages that may train future models, one builds the index behind AI search answers, and one fetches a page because a person asked about it. Each one reads its own lines in your robots.txt.

So one rule rarely does what people expect. Block GPTBot and you have opted out of OpenAI's model training, but ChatGPT search can still show your pages. Block everything with "AI" in its name and you may disappear from the answers your buyers read.

This guide sorts the bots by kind, lists each with the purpose its company documents, gives you a robots.txt to copy and shows how to check your logs. We checked every documentation page on 29 October 2026. Bots change, so check the linked pages before you rely on a rule.

Training bots, search bots and user fetchers

Training crawlers collect pages that may be used to train AI models. Blocking one is how you opt out of that company's training. Where a company also runs a separate search crawler, the two are controlled separately: OpenAI says "each setting is independent of the others", and Mistral says its training crawler "is not used for search indexing or to answer live user queries."

Search crawlers build the index an assistant searches when it answers with sources. Blocking one can keep you out of that company's AI answers. OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." For Google, Microsoft and Apple, the normal search crawler does this job.

User-triggered fetchers load one page because a person asked, for example by pasting your link into a chat. They don't crawl, and several companies say robots.txt may not apply because a person started the request. OpenAI says of ChatGPT-User that "robots.txt rules may not apply." Perplexity says Perplexity-User "generally ignores robots.txt rules." Google says its user-triggered fetchers "generally ignore robots.txt rules", and Meta says Meta-ExternalFetcher "may bypass robots.txt rules." Anthropic and Mistral differ. Anthropic says its bots honour robots.txt and that Claude-User lets site owners "control which sites can be accessed through these user-initiated requests." Mistral says MistralAI-User "governs which sites these user requests can be made to."

Control tokens are names you can use in robots.txt that don't belong to a crawler. Google-Extended and Applebot-Extended tell Google and Apple how they may use pages their normal crawlers fetched. Google says Google-Extended "doesn't have a separate HTTP request user agent string", and Apple says Applebot-Extended "does not crawl webpages."

Mixed-purpose crawlers do more than one job under one name. Meta says Meta-ExternalAgent crawls "for use cases such as training foundation AI models or improving products by indexing content directly." Amazon says Amazonbot "is used to improve our products and services" and "may be used to train Amazon AI models." Blocking one of these blocks every job it does.

The current AI bot list

Every row comes from the company's own page, linked in the last column. We left out crawlers whose operators don't publish a page describing them.

A few details in those pages change how your rules behave:

• Microsoft has no separate AI crawler. Bing's webmaster guidelines say "Bing and Copilot search experiences rely on the same core crawling, indexing, and ranking foundation as traditional search." Microsoft's controls for AI use are page tags instead. In its 2023 announcement, written when Copilot was called Bing Chat, content tagged NOARCHIVE is left out of chat answers and model training, and content tagged NOCACHE can appear with only its URL, title and snippet. Both still appear in Bing search results.
• Applebot copies Googlebot's rules. Apple says that if your robots.txt doesn't mention Applebot but mentions Googlebot, Applebot "will follow Googlebot instructions." Amazon's Amzn-SearchBot does the same with rules you give other search bots.
• Google-Extended doesn't touch Search. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." But it does cover grounding in Gemini Apps, so blocking it can keep your pages out of grounded answers there.

A robots.txt to copy

One rule from the standard, RFC 9309, catches almost everyone. A crawler follows the group that names it, and only uses the User-agent: * group "if no matching group exists." So any bot you name gets none of the rules under *. If you keep crawlers out of private paths, repeat those lines in every named group.

This version allows AI search and pages people ask for, and blocks training:

# AI search: allowed, except private paths
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Meta-WebIndexer
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: DuckAssistBot
Allow: /
Disallow: /admin/

# Pages a person asked for: allowed, except private paths
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
Allow: /
Disallow: /admin/

# Model training and open datasets: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: MistralAI-Training
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot, Bingbot and Applebot
User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Check the trade-offs before you paste it:

• Google-Extended also covers grounding in Gemini Apps, as above.
• Meta-ExternalAgent and Amazonbot do more than training, so blocking them blocks their other jobs too.
• CCBot feeds an open dataset. Blocking it keeps your future pages out of Common Crawl, whoever would have used them.

To block AI search as well, move the search and user fetcher names into the blocked group. Expect fewer citations in AI answers, and remember that some user fetchers may not follow the rule. To allow everything, you need no AI rules at all: RFC 9309 says that when no group matches and there is no * group, "no rules apply."

robots.txt also works per host. Anthropic asks you to add rules "for every subdomain that you wish to opt out from", and Amazon reads robots.txt for each host separately. A rule on www.example.com doesn't cover docs.example.com.

How long a change takes

How robots.txt relates to llms.txt

The two files do different jobs, and you can use both.

robots.txt says which URLs a crawler may fetch. Crawlers that follow it, follow it; nothing makes them. RFC 9309 is blunt: "These rules are not a form of access authorization."

llms.txt summarises your site for AI agents and points them to your most useful pages. It controls nothing. Our llms.txt guide covers the format and who reads it.

They interact in one way: a bot you disallow from your whole site can't fetch your llms.txt either, because only /robots.txt itself is "implicitly allowed." And neither is security. RFC 9309 warns that "listing paths in the robots.txt file exposes them publicly." Put private pages behind a login.

For finer control, some companies read tags on the page. Apple won't use content tagged nosnippet "as additional context and up-to-date content when AI models are used to generate output" in its products. Bing uses NOARCHIVE and NOCACHE, as above.

A shared AI rule for robots.txt is being drafted but isn't here yet. The IETF's AI Preferences working group has proposed a Content-Usage line, such as Content-Usage: train-ai=n, that says how content may be used rather than whether it may be fetched. When we checked, it was still an Internet-Draft, and none of the crawler pages above mentioned it.

Checking your server logs

Your access logs show which bots really visit and what they fetch.

Count visits by AI user agent. With a standard combined log format, this lists each AI user agent and how often it appears:

grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Agent|meta-|Amazonbot|Amzn-|MistralAI|DuckAssistBot|CCBot|Applebot|bingbot" access.log \
  | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

See what one bot fetched, for example the paths OAI-SearchBot requested most:

grep "OAI-SearchBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check that it's real. User agents are easy to fake. Google warns that "the HTTP user agent string can be spoofed", and Common Crawl says it is "aware of crawlers falsely identifying themselves as CCBot." There are two ways to check:

1. Reverse, then forward DNS. Google's verification steps: run host on the IP from your log, check the name ends in googlebot.com, google.com or googleusercontent.com, then run host on that name and confirm it returns the same IP. Applebot resolves to applebot.apple.com and CCBot to crawl.commoncrawl.org.
2. Published IP ranges. OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, Mistral, DuckDuckGo and Common Crawl publish IP lists, linked from the pages in the table. Bing offers a Verify Bingbot tool.

Look for two kinds of problem. First, a verified bot fetching paths you disallowed, after the times in the table above have passed. Several of the documentation pages list a contact address for reports. Second, bots you want that never arrive, often because a firewall or CDN rule blocks them. Perplexity's page explains how to allow its bots in Cloudflare and AWS WAF. Anthropic adds a warning for anyone tempted to block by IP: it "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."

How BeforeQuery helps

The SEO Agent in BeforeQuery's AI Marketer checks whether your robots.txt lets ChatGPT, Claude, Perplexity, Google and Bing search crawlers in, and whether your page can be read without JavaScript. It also checks your robots.txt, sitemap.xml and llms.txt. Your first homepage audit is free. On paid plans it audits every day and writes 2 fixes a day, each with code to paste. On Starter and Growth, the Coding Agent can put robots.txt changes, rules for AI crawlers and an llms.txt built from your site into a GitHub pull request. It never merges; you review and merge.

Letting the right crawlers in is the first step of generative engine optimisation. The GEO Agent asks ChatGPT, Gemini, Perplexity and Claude the questions your buyers ask and records whether you are named, and its GEO score includes how ready your site is for AI crawlers. Sign up free to see where you stand.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI says its robots.txt settings are independent of each other. GPTBot covers training and OAI-SearchBot covers ChatGPT search, so you can allow OAI-SearchBot and disallow GPTBot to appear in ChatGPT search while opting out of training. Sites that block OAI-SearchBot aren't shown in ChatGPT search answers, though they can still appear as navigational links.

Does blocking Google-Extended keep my site out of AI Overviews?

No. Google says Google-Extended does not affect a site's inclusion in Google Search, and AI Overviews and AI Mode are part of Search, which Googlebot crawls. Google-Extended controls whether your content is used to train Gemini models and to ground answers in Gemini Apps and Vertex AI.

Do AI crawlers obey robots.txt?

The companies in our table say their automatic crawlers do. User-triggered fetchers are different: OpenAI, Perplexity, Google, Meta and Amazon say theirs may not follow robots.txt, because a person asked for the page. robots.txt is a request, not a lock. The standard itself says its rules are not a form of access authorisation.

How long does a robots.txt change take to work?

It varies: about 24 hours for OpenAI, Perplexity and Meta, 72 hours for DuckAssistBot, and Amazon may use a cached copy up to 30 days old. The robots.txt standard says crawlers should not use a cached copy for more than 24 hours.

Should I block AI training crawlers?

That's a business decision. Blocking training crawlers doesn't remove you from AI search results, as long as you allow the separate search crawlers. But Google-Extended, Meta-ExternalAgent and Amazonbot each do more than training, so read each company's page before you block.

Is there one robots.txt line that opts out of all AI training?

Not yet. Each company uses its own name, so you list them one by one. The IETF is drafting a Content-Usage rule for robots.txt that would state AI usage preferences in one line, but when we checked it was still an Internet-Draft, not a standard.

Sources

• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Google: List of Google's common crawlers
• Google: List of Google user-triggered fetchers
• Google: Verify requests from Google crawlers and fetchers
• Bing Webmaster Tools: Overview of Bing crawlers
• Bing Webmaster Tools: Webmaster guidelines
• Bing Webmaster blog: New options to control usage of content in Bing Chat (September 2023)
• Apple: About Applebot
• Perplexity: Perplexity crawlers
• Meta: Meta web crawlers
• Amazon: Amazon web search crawlers
• Mistral: Mistral crawlers
• DuckDuckGo: Is DuckAssistBot related to DuckDuckGo?
• Common Crawl: CCBot
• RFC 9309: Robots Exclusion Protocol
• IETF: Associating AI usage preferences with content in HTTP (Internet-Draft)