AI crawler checker. Which AI bots may read your site?

Type a website to see whether its robots.txt lets in the crawlers of ChatGPT, Claude, Perplexity, Google, and the AI labs, and the exact rule that decides for each.

Free, with no account. It reads robots.txt and asks no AI.Check whether AI recommends you

What the AI crawler checker shows.

  • Every AI crawler

    Fifteen crawlers, from ChatGPT search and Perplexity to GPTBot, ClaudeBot, and Google-Extended.

  • The rule that decides

    Whether a group names the crawler or covers every crawler, and the line that settles it.

  • What blocking costs

    Search crawlers decide whether AI answers can cite you. Training crawlers only feed model training.

How it works.

  1. 1

    Type a website

    Any public site, yours or a competitor’s.

  2. 2

    robots.txt is read

    Fetched the way a crawler fetches it, then read for each crawler’s own group or the group for every crawler.

  3. 3

    See who is kept out

    Blocked search crawlers first, since those cost citations, then the training crawlers.

Questions about the AI crawler checker.

Should I block GPTBot or ClaudeBot?

That is your choice. GPTBot, ClaudeBot, Google-Extended, and CCBot collect text to train models, and blocking them costs no citations. Blocking OAI-SearchBot, Claude-SearchBot, or PerplexityBot stops those engines from citing your pages.

What is the difference between a search crawler and a training crawler?

A search crawler fetches pages so an AI answer can cite them today, as OAI-SearchBot does for ChatGPT search. A training crawler gathers text that may be used to train a future model, as GPTBot does. Each company names them separately so a site can allow one and block the other.

My robots.txt does not name any AI crawler. Is that a problem?

No. A crawler with no group of its own follows the group for every crawler, the one under User-agent with a star. If that group allows your pages, every AI crawler may read them.

Is the AI crawler checker free?

Yes, with no account. It reads robots.txt and asks no AI. A site checked in the last fifteen minutes shows the same result.

Keep going.