Let AI crawlers read your site
Which AI crawlers decide whether your pages can be cited, which only train models, and how to check and monitor your robots.txt, sitemap, and llms.txt.
On this page
AI crawlers come in two kinds. Search crawlers fetch pages so an AI engine can cite them in answers, and blocking one keeps your site out of that engine's answers. Training crawlers only gather text to train models, and blocking them costs no citations. Your robots.txt decides which of them may read your site.
Search crawlers and training crawlers#
These are the crawlers Behavior checks for, in robots.txt, on the free AEO checker and in site monitoring.
Search crawlers. Allow these, or the engine cannot read your pages to cite them.
| Crawler | Company | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Reads pages for ChatGPT search |
| ChatGPT-User | OpenAI | Fetches a page when a person's question in ChatGPT needs it |
| Claude-SearchBot | Anthropic | Reads pages for Claude's search |
| Claude-User | Anthropic | Fetches a page when a person's question in Claude needs it |
| PerplexityBot | Perplexity | Reads pages for Perplexity answers |
Training crawlers. Blocking these is a fair choice. They gather text to train models and do not decide citations.
| Crawler | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects training data for OpenAI models |
| ClaudeBot | Anthropic | Collects training data for Anthropic models |
| Google-Extended | Controls whether Google uses your pages to train its AI | |
| CCBot | Common Crawl | Builds the open Common Crawl dataset many models train on |
Google's own AI answers, AI Mode and AI Overviews, read pages through Googlebot, the same crawler as Google Search. Google's crawler documentation says Google-Extended does not affect a site's inclusion in Google Search. The checks here do not test Googlebot.
What to allow in robots.txt#
Here is a robots.txt that lets every search crawler in, keeps all crawlers out of private paths, and opts out of training.
User-agent: *
Allow: /
Disallow: /account/
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
Disallow: /account/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
Sitemap: https://yoursite.com/sitemap.xml
If you are happy for models to train on your pages, leave the training group out. The one mistake to avoid is a blanket Disallow: / for every agent, often left over from a staging site, which shuts out every search crawler too.
Sitemap and llms.txt#
A sitemap lists your pages so crawlers find the ones your menus do not link. Name it in robots.txt, as in the example.
An llms.txt file is an optional plain summary of your site and its key pages for AI tools. It is a proposal, not a crawler rule. What is llms.txt, and do you need one? covers how to write one.
Pages AI can read without JavaScript#
Vercel's analysis of AI crawler traffic (December 2024) found that the major AI crawlers, GPTBot and ClaudeBot among them, do not run JavaScript. A page whose text only appears after scripts run can arrive nearly empty. Check that your important pages show their text, title, description, and headings in the HTML the server sends.
Check once, free#
The free AEO checker needs no account. Type a website and it reads robots.txt for the five search crawlers and four training crawlers above, then up to twenty pages without JavaScript. It checks titles, descriptions, headings, structured data, canonical links, broken links, llms.txt, the sitemap, and how easily AI answers can quote each page. Then it tells you what to fix first.
Monitor a site#
A site can change under you. A deploy can block crawlers, or a page can stop loading. Monitor a site crawls your site on a schedule (daily, weekly, or only when you ask), the way AI search reads it, and keeps the findings.
Its alerts tell you when something changes.
- The site score drops by the points you choose.
- A serious problem appears, such as broken links, pages returning errors, blocked AI crawlers, or anything critical that was not there before.
- A tracked page goes down, or comes back.
- Google search clicks fall, week on week, when Search Console is connected and the week before had fifty clicks or more.
Alerts go to a Slack or Discord channel, or by email, and every alert is kept in the monitor's log. AI visibility alerts and client reports covers setting them up.
Then see if AI names you#
Letting crawlers in makes your pages eligible to be cited. It does not decide who answers recommend. Run the free check to see what Google AI Overviews, AI Mode, and an AI model say about your brand, or compare plans to track ChatGPT and Gemini too, daily.
Related reading#
Updated Nov 9, 2026