What is an AI crawler?
An AI crawler is a bot run by an AI company that fetches web pages to train models, build an AI search index or answer a user's question.
Last updated 29 October 2026
Three kinds of AI crawler
- Training crawlers collect pages that may be used to train AI models, such as OpenAI's GPTBot and Anthropic's ClaudeBot.
- Search crawlers build the index an assistant searches when it answers, such as OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot.
- User-triggered fetchers visit a page because someone asked the assistant a question, such as ChatGPT-User, Claude-User and Perplexity-User.
Managing them with robots.txt
- Each name gets its own rules, so you can allow search crawlers and block training crawlers.
- Blocking a search crawler can keep you out of that assistant's answers. OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links.
- User-triggered fetchers may not follow robots.txt. OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.
- Google-Extended isn't a separate crawler but a robots.txt name. It controls whether Google may use your content to train Gemini models and for grounding, and Google says it doesn't affect inclusion or ranking in Google Search.
- robots.txt is a request, not a lock. The standard says its rules aren't a form of access authorisation.
Checking your own site
Open yoursite.com/robots.txt to see what each crawler may fetch, and search your server logs for these names to see which ones visit. A firewall or bot filter can block AI crawlers even when robots.txt allows them; Perplexity, for example, says you may need to allow its bots explicitly in your firewall.
See how BeforeQuery's AI employees do this work, with a person approving anything that matters.
Start freeQuestions
Should I block AI crawlers?
It depends on what you want. To appear in AI answers, allow the search crawlers and user-triggered fetchers. You can still block the training crawlers, because each one has its own rules.
Does blocking GPTBot keep me out of ChatGPT?
No. GPTBot is OpenAI's training crawler. Appearing in ChatGPT search answers depends on OAI-SearchBot, which you allow or block with its own rule.
How quickly do robots.txt changes take effect?
OpenAI and Perplexity both say a change can take up to about 24 hours to reach their systems.
Does BeforeQuery check my robots.txt?
Yes. The AI Marketer's SEO Agent checks whether your robots.txt lets the ChatGPT, Claude, Perplexity, Google and Bing search crawlers in, and whether your page can be read without JavaScript.
Hire your first AI employee
The AI Marketer is ready today, free to start. For your other teams, tell us which one needs an AI employee and we'll show you how it works with your tools.