What the checker actually reads
When you submit a domain, the tool fetches two things live: your homepage and your robots.txt. It does not use a cached index, and it does not need access to your hosting, your CMS, or your analytics. Everything it evaluates is public.
From those responses it evaluates three public directive surfaces for each named agent. These surfaces behave differently:
- robots.txt — a Disallow rule naming the bot, or a blanket rule that catches it. This is the most common cause and the easiest to fix.
- meta robots — a noindex tag in the HTML of the page itself. The crawler is allowed to fetch the page but told not to use it.
- X-Robots-Tag — the same instruction delivered as an HTTP header rather than a tag. Invisible when you view the page source, which is why it survives so long.
A page can be perfectly written, fast, and well-structured and still be invisible, because access is decided before any of that is read.
The distinction that matters most: training bots vs search bots
A site owner may reasonably choose not to allow training collection while still allowing search or user-initiated retrieval. Those are different policy choices, so the checker keeps the detector's purpose label beside every agent.
The purpose label describes how the Revvye detector classifies the agent: training, search, user-initiated, or advertising. A detected block closes that declared access path; it does not by itself prove what any external product indexed, cited, or displayed.
The table below is generated from the same named-agent catalog used by the detector. For current policy consequences beyond the observed directive, confirm the agent's purpose in the operator's own documentation before changing access rules.
| Named agent | Operator | What Revvye reports |
|---|---|---|
| OAI-SearchBot | OpenAI | Classified by the Revvye detector as search; the result reports only whether a public directive blocks that named agent. |
| GPTBot | OpenAI | Classified by the Revvye detector as training; the result reports only whether a public directive blocks that named agent. |
| ChatGPT-User | OpenAI | Classified by the Revvye detector as user-initiated fetch; the result reports only whether a public directive blocks that named agent. |
| ClaudeBot | Anthropic | Classified by the Revvye detector as search; the result reports only whether a public directive blocks that named agent. |
| anthropic-ai | Anthropic | Classified by the Revvye detector as training; the result reports only whether a public directive blocks that named agent. |
| PerplexityBot | Perplexity | Classified by the Revvye detector as search; the result reports only whether a public directive blocks that named agent. |
| Perplexity-User | Perplexity | Classified by the Revvye detector as user-initiated fetch; the result reports only whether a public directive blocks that named agent. |
| Google-Extended | Classified by the Revvye detector as training; the result reports only whether a public directive blocks that named agent. | |
| Googlebot | Classified by the Revvye detector as search; the result reports only whether a public directive blocks that named agent. | |
| Bingbot | Microsoft | Classified by the Revvye detector as search; the result reports only whether a public directive blocks that named agent. |
| AdsBot-Google | Classified by the Revvye detector as advertising; the result reports only whether a public directive blocks that named agent. | |
| DuckAssistBot | DuckDuckGo | Classified by the Revvye detector as search; the result reports only whether a public directive blocks that named agent. |
| MistralAI-User | Mistral | Classified by the Revvye detector as user-initiated fetch; the result reports only whether a public directive blocks that named agent. |
| YouBot | You.com | Classified by the Revvye detector as search; the result reports only whether a public directive blocks that named agent. |
Reading your result
The headline number is how many of the 14 named agents were blocked by a detected public directive. Each row preserves the agent, operator, purpose classification, observed source, and allowed/blocked state returned by the live API.
- Zero blocked — no tested public directive denied the named agent set in this fetch. That does not prove indexing, citation, or recommendation.
- Only training agents blocked — this may reflect an intentional content-use policy. Confirm the decision with the site owner and the current operator documentation.
- Search or user-initiated agents blocked — review the matched public directive and decide whether that access path should be closed.
- No robots.txt at all — crawlers default to allowed, so this is not an error. It does mean you have no way to express a preference later without adding the file.
If the homepage fetch itself fails, the meta and header checks are limited — the tool says so explicitly rather than reporting a clean pass it cannot support. robots.txt rules are still evaluated in that case, because robots.txt is fetched separately.
Fixing a block you did not intend
robots.txt lives at the root of your domain and is plain text. Whoever maintains your site can edit it in a few minutes. The change is to remove or narrow the Disallow rule under the user-agent you want to admit. Longer, more specific rules win over broader ones, so you can open a single path without opening the whole site.
A meta robots noindex is edited in the page template or a page-level publishing setting. An X-Robots-Tag is delivered at the server or CDN layer. The live tool does not impersonate every named agent at a network edge, so agent-specific WAF behavior remains outside this result and needs separate infrastructure inspection.
Change one thing at a time and re-run the check. When the public fetch succeeds, the checker immediately repeats the same directive evaluation and exposes the new result.
What this tool does not tell you
Being readable is a precondition for being cited, not a cause of it. This check confirms the door is open. It does not tell you whether an assistant currently mentions your business, whether your pages state what you do in a form worth quoting, or how you compare to the competitor that is being cited instead.
It also covers one part of the 8-category public scan. The published taxonomy currently contains 43 checks across crawler access, structured data, markup, metadata, privacy, security, email posture, and technology signals. The scanner starts from public pages and moves private results into the email-linked flow.
Common questions
01How do I know if my site is blocking GPTBot?+
Enter your domain above. The result lists GPTBot by name, preserves its training-purpose classification from the detector catalog, and reports the public directive that allowed or blocked it. Confirm broader product effects in OpenAI's current documentation before changing policy.
02Does blocking AI crawlers hurt my Google rankings?+
The checker keeps Google-Extended and Googlebot as separate named agents with separate purpose labels. It reports the public directive state; it does not measure search ranking impact. Confirm current product consequences in Google's documentation before changing either rule.
03I have no robots.txt. Is that a problem?+
No. Crawlers treat a missing robots.txt as permission to read everything, so an absent file is a default-allow state. The only downside is that you have no mechanism to express a preference until you add one.
04Why does the checker say a bot is blocked when my robots.txt looks fine?+
Because robots.txt is one of three directive surfaces this tool reads. A homepage noindex meta tag or X-Robots-Tag header can produce a blocked result even when robots.txt allows the agent. Agent-specific firewall behavior is outside this public result.
05Does this check whether ChatGPT actually mentions my business?+
No. This tool checks access — whether the crawlers are permitted to read your pages. Whether an assistant chooses to mention or cite you is a separate question that depends on how legible and specific your content is once it has been read.
06Is the checker free, and does it need my email?+
It is free and it does not ask for an email, an account, or a card. It fetches two public documents from your domain and reports what they say.