Quick answer
A robots.txt tester reads a site's robots.txt file and works out whether a given crawler may fetch a given URL, using the same rules Google does. Enter a URL above to see the verdict for Googlebot, Bingbot and the main AI crawlers, the rule that decided it, and any problems in the file.
Enter the address of any page. The tester fetches the robots.txt file for that site, finds the group of rules that applies to each crawler, and decides whether the crawler may fetch your page. It shows the rule that made the decision, the sitemaps the file lists, and warnings about lines search engines ignore or misread. Add a user agent name to test one crawler only. It is free, needs no account and allows 30 checks an hour.
How robots.txt rules are read
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules. Google reads it like this, and the tester follows the same logic:
- Pick the group. A crawler uses the group with the most specific user agent that matches its name. Googlebot uses a
Googlebotgroup if there is one, and the*group only if there isn't. Groups that name the same crawler are combined. - Find the matching rules. Each rule is a path prefix.
*matches any characters and$marks the end of the URL, soDisallow: /*.pdf$blocks every URL ending in .pdf. - Let the longest rule win. When several rules match, the most specific (longest) one decides. If an
Allowand aDisallowrule are equally long,Allowwins. - No rule means allowed. A URL that no rule matches may be crawled.
This matters because the order of lines doesn't decide anything. A short Disallow: /shop and a longer Allow: /shop/sale let crawlers into the sale pages and keep them out of the rest of the shop, whichever comes first in the file.
What the robots.txt tester results tell you
For each crawler the tester shows allowed or blocked, the group it used, and the rule that matched. Blocked results are worth a second look: is that intended? A block on /cart or /search is normal. A block on a product category or a blog section usually isn't.
Remember what robots.txt does and doesn't do. It controls crawling, not indexing. A blocked page can still appear in search results as a bare link if other sites link to it, because Google can't fetch the page to see a noindex. To keep a page out of search, allow crawling and use a noindex meta tag or header instead.
The AI crawlers in the default list behave differently from search engines, and blocking some of them affects whether AI assistants can cite you. Our AI crawler access checker explains each one and which to allow.
Common robots.txt mistakes
Blocking the whole site. User-agent: * followed by Disallow: / keeps every compliant crawler out. It is often left over from a staging site. The tester flags it.
Blocking CSS and JavaScript. Google renders pages like a browser. If it can't fetch the files that build the page, it may see an empty or broken layout.
Using robots.txt to hide pages. As above, a blocked URL can still be indexed. Use noindex for that.
Unsupported lines. Google ignores Crawl-delay (Bing respects it) and doesn't support Noindex: inside robots.txt at all.
No Sitemap line. Listing your sitemap in robots.txt helps every search engine find it, not only the ones you submit it to.
A server error. If robots.txt answers with a 5xx error, Google treats the whole site as off limits until it recovers. A 404 is fine: it means no rules, so everything may be crawled.
Testing changes safely
Before you edit a live robots.txt, test the URLs that matter most: your homepage, a product or article page, a category page and anything you intend to block. After publishing, run them again. Google usually picks up a changed file within a day.
A site-wide audit catches the other half of the problem: pages that are crawlable but marked noindex, sitemap URLs that robots.txt blocks, and blocked resources. Seotific's full audit checks all of them alongside the page itself.
Check your own page
Seotific audits a URL against the keyword it should rank for and lists the fixes in order of impact. Free plan, no card.
Run a free auditFrequently asked questions
What is a robots.txt tester?
A robots.txt tester is a tool that reads a site's robots.txt file and tells you whether a particular crawler, such as Googlebot or Bingbot, is allowed to fetch a particular URL, and which rule in the file decides it.
Does robots.txt stop a page from being indexed?
No. It stops compliant crawlers from fetching the page, but the URL can still be indexed from links elsewhere. Use a noindex meta tag or HTTP header, on a page that crawlers may fetch, to keep it out of results.
Do AI crawlers obey robots.txt?
The main ones say they do, including GPTBot, ClaudeBot and PerplexityBot. Some user-triggered fetchers, which load a page because a person asked an assistant about it, say robots.txt may not apply to them.
Why does my rule not work for Googlebot?
Usually because Googlebot matched a more specific group. If the file has a Googlebot group, Googlebot ignores the rules under User-agent: * entirely, so repeat any shared rules in its own group.