Quick answer
An AI crawler access checker reads your robots.txt and reports, for each AI crawler, whether it may fetch a given URL. Allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want to be cited in AI answers; blocking training crawlers such as GPTBot and ClaudeBot is a separate choice.
Enter a URL above and the checker tests it against your robots.txt for 13 AI crawlers: OpenAI's three, Anthropic's three, Perplexity's two, Google-Extended, Applebot-Extended, Common Crawl's CCBot, ByteDance's Bytespider and Amazonbot. Each gets an "Allowed" or "Blocked" verdict for that exact URL, using the same robots.txt matching rules the crawlers apply.
This page explains how to read those verdicts, which crawlers you probably want to allow, and the layer robots.txt can't show you: your firewall. It sits under the main AI search readiness checker guide.
Why AI crawler access needs checking now
For most of the web's history, "can search engines crawl my site?" meant Googlebot and Bingbot. Now each AI company runs several crawlers for different jobs, adds new ones every year or so, and documents them in different places. A robots.txt written in 2023 may not name a single crawler that matters in 2026.
At the same time, more sites block AI crawlers than their owners realise. Cloudflare started blocking AI training crawlers by default on new domains in July 2025, and from 15 September 2026 new Cloudflare domains block bots classed as Training or Agent on pages that display ads. "Block AI bots" templates for robots.txt circulate widely and often include the search crawlers along with the training ones. WordPress security plugins and hosting firewalls add their own rules.
The result: sites that want to be cited in ChatGPT, Claude or Perplexity answers sometimes block the exact crawler that would make that possible.
Insight: Name each crawler you care about in robots.txt, even if your User-agent: * group already allows everything. It documents the decision, and it protects you if someone later adds a broad Disallow to the wildcard group.
The 13 AI crawlers the checker tests
| User agent | Company | Job | What blocking it does |
|---|---|---|---|
| GPTBot | OpenAI | Collects training data | Keeps content out of OpenAI model training |
| OAI-SearchBot | OpenAI | Indexes for ChatGPT search | Removes pages from ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches when a user asks | OpenAI says robots.txt may not apply |
| ClaudeBot | Anthropic | Collects training data | Keeps content out of Claude training |
| Claude-SearchBot | Anthropic | Indexes for Claude's search | May reduce visibility in Claude's answers |
| Claude-User | Anthropic | Fetches when a user asks | Claude can't retrieve the page for users |
| PerplexityBot | Perplexity | Indexes for Perplexity answers | Pages aren't indexed by Perplexity |
| Perplexity-User | Perplexity | Fetches when a user asks | Generally ignored, per Perplexity |
| Google-Extended | Gemini training and grounding control | No effect on Search or AI Overviews | |
| Applebot-Extended | Apple | Apple Intelligence training control | No effect on Applebot's search crawling |
| CCBot | Common Crawl | Open web dataset | Keeps content out of Common Crawl, used by many models |
| Bytespider | ByteDance | Training data | Keeps content out of ByteDance training (compliance has been questioned) |
| Amazonbot | Amazon | Alexa and Amazon AI features | Keeps content out of Amazon's AI features |
Google-Extended and Applebot-Extended are tokens rather than separate crawlers: Googlebot and Applebot do the fetching, and the token tells them how the content may be used. That is why blocking Google-Extended doesn't remove you from AI Overviews.
Each of the three most-asked-about crawlers has its own page with copy-paste rules: GPTBot robots.txt, ClaudeBot robots.txt and PerplexityBot robots.txt.
How to check AI crawler access in 6 steps
- Run the checker above on your homepage and on one URL from each important section, such as
/blog/and/products/. - For any crawler marked Blocked, open your robots.txt and find the group that names it, or the
User-agent: *group if none does. - Decide per crawler whether the block is intended, using the table above and your goal for that AI product.
- Check your CDN or host for AI bot settings (Cloudflare: Security and AI Crawl Control; Vercel and others: firewall rules) and make them match your robots.txt.
- Search your server or CDN logs for the user agents you allow and confirm they receive status 200, not 403.
- Re-run the checker after any change, and allow about 24 hours before expecting OpenAI's search systems to reflect it.
How robots.txt matching works for AI crawlers
The checker applies the standard rules, which every major crawler follows:
- A crawler obeys only the most specific group that names it. If OAI-SearchBot has its own group, it ignores the
User-agent: *group entirely, including anyDisallowlines there. - Several user agents can share a group by listing multiple
User-agent:lines above the rules. - The longest matching path wins between
AllowandDisallow.Allow: /blog/beatsDisallow: /for blog URLs. - User agent names are matched case-insensitively, but use the documented spelling to avoid confusion.
- No robots.txt means everything is allowed. A robots.txt that returns 5xx can cause crawlers to stop crawling until it recovers.
This is why the checker tests a specific URL rather than the whole site: the answer genuinely differs by path.
Check your own page
Seotific audits a URL against the keyword it should rank for and lists the fixes in order of impact. Free plan, no card.
Run a free auditWhich crawlers to allow, by goal
| Your goal | Allow | Block | Notes | Seotific's role |
|---|---|---|---|---|
| Maximum AI visibility | All 13 | None | The default if you have no robots rules | Confirms nothing blocks them |
| Cited in AI answers, no training | OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot | GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended, Applebot-Extended | The most common choice for publishers | Shows each crawler's status separately |
| Paywalled or private content | None on private paths | All on private paths | Use authentication; robots.txt is a request, not a lock | Flags private paths that are crawlable |
| Out of AI products entirely | None | All AI crawlers, plus firewall rules | User-triggered fetchers may still get through | Confirms the rules are in place |
The layer robots.txt can't show: firewalls and CDNs
The checker reads robots.txt, which is a set of instructions crawlers choose to follow. A firewall acts before any of that: it can return 403 to a crawler based on its user agent or IP range, and robots.txt never comes into play.
Three places to look:
- Cloudflare. The AI Crawl Control area lists AI crawlers by name with allow or block per crawler, plus the one-click "block AI bots" setting and an option to have Cloudflare manage your robots.txt. If either is on, Cloudflare may block or add rules your own robots.txt doesn't show.
- Hosting firewalls and WAFs. Rules that block "unknown bots" or rate-limit aggressive crawling often catch AI crawlers.
- WordPress security plugins. Several have bot-blocking lists that include AI crawlers.
To test, request a page with a crawler's user agent (curl -A "OAI-SearchBot" https://yoursite.com/). If that returns 403 and a normal request returns 200, a user-agent rule is blocking it. Some firewalls also verify crawlers by IP, so a spoofed request may be blocked even when the real crawler isn't; your logs are the final word. OpenAI and Perplexity publish their crawlers' IP ranges for exactly this purpose.
Telling real crawlers from impostors
Anyone can send a request with GPTBot in the user agent, and scrapers often do, hoping to slip through allow-lists. If your logs show heavy traffic from a "GPTBot" that ignores your robots.txt, check the IP addresses before blaming OpenAI. Requests from outside the company's published ranges are not the real crawler, and blocking them by IP at the firewall is reasonable. Google's crawlers can be verified with a reverse DNS lookup that resolves to googlebot.com or google.com.
Tools to use alongside this AI crawler access checker
This checker for robots.txt rules per crawler and URL. Your CDN's bot analytics (Cloudflare's AI Crawl Control lists requests by crawler) and server logs for what actually arrives. Each company's crawler documentation (OpenAI, Anthropic, Perplexity, Google) for current user agent names, which change.
Seotific's full audit includes AI crawler access as one check on every audited page, alongside whether the content is in the raw HTML and whether the page answers its question directly, because access alone doesn't get a page cited. The llms.txt checker covers the optional llms.txt file the same tool reports on.
Frequently asked questions
How do I check if AI crawlers can access my website?
Enter a URL in the checker above. It reads your robots.txt and reports whether each of 13 AI crawlers may fetch that URL. Then check your CDN or firewall settings and server logs, because a network-level block won't appear in robots.txt.
Should I block AI crawlers?
Block training crawlers (GPTBot, ClaudeBot, CCBot) if you object to your content training AI models; it has no effect on Google rankings. Think carefully before blocking search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), because that removes you from those products' answers and the traffic they send.
Why does the checker say a bot is allowed but I never see it in my logs?
Either the crawler hasn't visited yet, or something blocks it before robots.txt matters, usually a CDN or firewall rule. Check Cloudflare's AI bot settings and your hosting firewall, and test with the crawler's user agent to see the status it receives.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and live page fetches use ChatGPT-User. To stay out of ChatGPT search results, block OAI-SearchBot; to opt out of training only, block GPTBot.
How long do robots.txt changes take to affect AI crawlers?
OpenAI says its search systems take about 24 hours to reflect a robots.txt change. Other crawlers re-read robots.txt on their own schedules, typically within a day. Check your logs after a day or two to confirm the new rules are being followed.
Do AI crawlers respect robots.txt?
The major search and training crawlers from OpenAI, Anthropic, Perplexity, Google and Apple say they do. OpenAI and Perplexity say their user-triggered fetchers may not. Some smaller or less transparent crawlers have been reported ignoring robots.txt, so use firewall rules where you need enforcement.
Check your own page
Seotific audits a URL against the keyword it should rank for and lists the fixes in order of impact. Free plan, no card.
Run a free audit