Skip to main content
RankBrain

Robots.txt Generator and Tester

Builds a robots.txt file rule by rule, and tests any address against it or against your live file.

Runs in your browser. The file or text you give it is not uploaded. Free, no account.

Finding the problem is the first half.

RankBrain checks your robots.txt on every audit and flags a rule that blocks pages you want found.

RankBrain works on websites built in code, through the AI tool you already use (Cursor, Claude Code, Codex, ChatGPT).

Check my site free

How it works

You need a robots.txt that does what you mean, and no way to know if it does. This tool builds the file rule by rule and tests any address against it, so you can see which group matches and which rule wins. You can also load the robots.txt of a live site and test addresses against it.

  1. 1

    Add groups of rules for all crawlers or for specific ones. Presets cover the main search crawlers and the AI crawlers, so you can allow or block them deliberately.

  2. 2

    The tester applies the published matching rules: a crawler obeys the most specific group for its name, the longest matching path wins, and Allow wins a tie.

  3. 3

    You can load the robots.txt of any live site and test addresses against it. That fetch is the only step that uses our server.

What it does not do

  • robots.txt controls crawling, not indexing. A blocked page can still appear in results as a bare link; use noindex to keep a page out.

  • It is a request, not a lock. Well-behaved crawlers follow it; others ignore it.

When a source is busy or a site does not answer, this tool says so. It never fills the gap with a guess.

Questions

Why does my page still show in Google when robots.txt blocks it?

robots.txt controls crawling, not indexing. A crawler blocked from fetching a page can still list it as a bare link, without a title or description, because it found the address somewhere else. To keep a page out of results, let it be crawled and add a noindex rule to the page itself.

What happens when an Allow and a Disallow rule match the same address?

The tester applies the published matching rules: the longest matching path wins, and Allow wins when the two paths are the same length. So Disallow: /private/ beats a shorter Allow rule, and Allow: /private/public/ beats Disallow: /private/. To open an exception inside a blocked folder, write the Allow rule with the longer path.

Can I test my rules against a site that is already live?

Yes. Paste the address and the tool fetches that site's robots.txt and tests addresses against it. That fetch is the only step that uses our server. The builder itself runs in your browser. Remember that robots.txt is a request, not a lock. Well-behaved crawlers follow it, others ignore it.

Tools that answer the next question

All 74 free tools