Robots.txt checker for Google and AI crawlers

How to read your result

Fix the first group first. If Googlebot is blocked, Google can’t read the page for Search, and that includes AI Overviews and AI Mode. If OAI-SearchBot is blocked, the page won’t show up in ChatGPT search answers. If ChatGPT can’t read you, it can’t cite you.

The middle group opens a page because a person asked for it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, so for those two, “Blocked” is only a request they may ignore. Claude-User is different: Anthropic says its bots honor robots.txt.

Which crawlers should you allow?

Let every crawler in the first group in. They’re how Google, Bing, ChatGPT, Claude, Perplexity and Apple find your pages before they can name you.

Training is your call. OpenAI, Anthropic and Apple each keep search and training on separate crawlers, so blocking GPTBot, ClaudeBot or Applebot-Extended doesn’t take you out of their answers.

Google-Extended is the one in the first group that looks like a training bot. Google says it “does not impact a site’s inclusion in Google Search”, but it decides whether Gemini can use your pages to train and to back its answers in Gemini Apps. Block it and you’re out of both.

Robots.txt isn’t a lock. The file is public and its rules are requests, so listing a private path tells everyone it exists. Keep private pages behind a login.

How the checker decides

1. It finds the crawler’s group. Names match whatever the case, and groups that name the same crawler are combined. A crawler with no group of its own follows the User-agent: * group, and with no * group it has no rules. Applebot follows Googlebot’s group when it has none of its own, as Apple documents.
2. It picks the rule. Of the Allow and Disallow lines that match the start of the page’s path, the longest one wins. On a tie, Allow wins. * stands for any characters and $ for the end of the URL. Paths are case-sensitive, so /Admin isn’t /admin, and the query string counts.
3. /robots.txt itself is always allowed.

The rules come from RFC 9309 and Google’s robots.txt spec.

Google reads the first 500 KiB of the file and ignores the rest. How your server answers when Google asks for the file matters too: a 404 and a 500 have opposite results.

A worked example

The checker opens with this file and the page /admin/help/billing. Take Googlebot.

1User-agent: *
2Disallow: /admin/
3Allow: /admin/help
4
5User-agent: GPTBot
6User-agent: CCBot
7Disallow: /
8
9Sitemap: https://example.com/sitemap.xml

Googlebot has no group of its own, so it follows the * group. Two lines match the start of the path: line 2 (/admin/, 7 characters) and line 3 (/admin/help, 11 characters). The longer one wins, so Googlebot is allowed. Try /admin/users and only line 2 matches, so it’s blocked. GPTBot and CCBot share the group on lines 5 to 7, so they’re blocked everywhere.

Blocked isn’t the same as not indexed

Robots.txt controls crawling, not indexing. Google can’t read a blocked page, but it may still list the URL in results without a snippet when other pages link to it. To keep a page out of Google, let Googlebot crawl it and add a noindex tag. Google ignores a Noindex line in robots.txt.

Being crawlable is only the first step. Then your pages have to be indexed, show up for the searches buyers make and get named when they ask AI. That’s the work we do for 30+ software companies. Our guide on SEO for startups covers what to do first. For ChatGPT, read our guide on getting named in ChatGPT. If your site renders in the browser, our Next.js page shows what GPTBot can miss.

FAQ