Robots.txt Analyzer - Check & Validate

Built and tested by Pavel Ahmed · Last updated October 5, 2026

What Is a Robots.txt Analyzer and What Does It Do?

What does this tool do?

It fetches the robots.txt file for a host, then parses it the way a crawler does. It splits the file into user-agent groups, counts the groups and the rules they hold, lists every sitemap, and reports the HTTP status the server returned, because that status largely decides whether the rules apply at all. It then tells you, for six well known crawlers, whether the site is fully open, partly restricted or blocked outright, and it flags syntax mistakes such as a path with no leading slash or a directive that search engines no longer support. When no usable file comes back, the crawler rows say so instead of guessing.

Why is this useful?

One misplaced line can hide an entire section of a site from search. A rule that blocks a folder of stylesheets or scripts stops Google rendering the page it is trying to index, and a stray Disallow: / left over from a staging server removes a whole site from crawling. Because robots.txt is fetched from a fixed location and generally cached by Google for up to 24 hours, mistakes can be slow to surface and slow to undo, so it is worth reading the file as a machine reads it rather than as you remember writing it.

Who uses this tool?

• SEO specialists confirming that money pages, images and rendering assets are all crawlable.

• Web developers checking crawl rules before a redesign, a CMS migration or a domain move goes live.

• Site administrators keeping admin paths, search result pages and staging copies out of the index.

• Agencies auditing a client site they have no server access to.

• Publishers deciding which AI crawlers, such as GPTBot or Google-Extended, may read their content.

How Do You Use This Tool?

1. Type a domain such as https://example.com into the Website address box. Any path you include is ignored, because robots.txt only ever lives at the root, and the site you are reading now is checked automatically when the page loads.

2. Press Analyze, or the Enter key. The tool requests /robots.txt from that host over https and reports the HTTP status it got back.

3. Read the summary, then the per crawler verdicts, then the checks list. Errors are shown before warnings and notes.

4. Enter a path such as /admin/settings.html in the path tester and switch crawler tabs to see which rule decides it. Enter works there too.

5. If a host cannot be read from a browser, open its file in a new tab, copy the text into the Paste a file tab, and analyze it there.

User-agent: *
Disallow: /search
Disallow: /admin/
Allow: /admin/help/
Sitemap: https://example.com/sitemap.xml

These are illustrative files on the reserved domain example.com. Each row was worked out line by line from this page's parser and verdict logic, not from live lookups.

How Do You Read the Results?

How the Analysis Is Worked Out

Every row in the report is derived in your browser from the file text alone. Nothing is looked up in a database and no crawler is contacted: the tool reconstructs the decision a crawler would make and shows you the steps it took, so where two rules could apply the row names the one that actually won. The work happens in seven steps, and each one removes something the next one no longer has to consider.

First, the input is reduced to a bare host

Whatever you type is trimmed of its scheme, then of any path, query string or fragment, then of any user:pass@ prefix. The order matters and is not the obvious one. Stripping the credentials first lets an @ inside the path swallow the real host, so example.com/@name would be read as a host of name. Taking the path off first avoids that entirely, and the request is always made against https://<host>/robots.txt — one path, one scheme, no exceptions.

Then the file is fetched twice over, if it has to be

The first attempt is a plain fetch from your browser directly to the target host. It is the fastest path and the most private one, because nothing leaves your machine. It also fails whenever the host does not send a cross-origin header permitting a page on another origin to read its files, which is the case for most robots.txt files. When the direct read fails the same request is retried once through the public AllOrigins relay, which returns the body along with the original HTTP status code; the domain you typed is visible to that service. If both attempts fail the page says so and points you at the Paste a file tab rather than inventing content.

The status decides whether a file is parsed

Parsing only happens on a 2xx response. Every other status is reported as its consequence, because the rules inside a 404 body, a redirect page or an error page are never read by a crawler and reporting them would be worse than reporting nothing:

• 2xx — the body is parsed and every section of the report is filled in from it.
• 404 or 410 — no file exists. The crawler rows say crawling is unrestricted, which is the fact, and the summary shows the status rather than a rule count that does not exist.
• 429 or any 5xx — Google first stops crawling the site and keeps retrying, then can fall back to its last good copy of the file. The crawler rows explain this instead of showing a rule verdict, because the status, not the file, drives the outcome at that point.
• Any other 4xx, including 401 and 403 — treated as though no file exists. The result is the same as a missing file: crawling is unrestricted. This is the case that catches people out, since a permissions error on the file hands the whole site to a crawler rather than locking it away.

When the server redirects, your browser follows the chain itself and the tool reports the status of the final response. If the chain is too long or the target refuses cross-origin reads, the direct read fails and the tool tries the relay once, then asks you to use the Paste a file tab. This is different from Google, which follows about five hops and then treats the file as missing.

The body is parsed one line at a time

Each line is handled in the same order. A UTF-8 byte order mark on the very first character is stripped and noted, because Google tolerates it but nothing guarantees it. A line whose first non-space character is # is counted as a comment and skipped. Any # later in a line cuts the rest off, so a trailing comment after a rule does not become part of the path. A blank line is skipped. Everything else is split at the first colon: the part before it is lowercased and used as the directive name, and the part after it is trimmed and used as the value.

A line with no colon at all is recorded as an error, because no crawler can read it. A known directive is dispatched to its own handler, and an unrecognised directive is recorded as a warning rather than breaking the parse, since crawlers themselves skip lines they do not recognise. The Host and Clean-param directives are noted as Yandex extensions, and Noindex is recorded as an error with an explanation, because it stopped working in 2019 and leaving it in place silently does nothing.

Groups are built by the same rule a crawler uses

A User-agent line opens a new group when an Allow or Disallow line has appeared since the previous agent line, or joins the current one when none has, which is how several consecutive agent lines share a single set of rules. Other lines, such as Sitemap, are ignored for grouping, as Google describes. Rules that appear before the very first agent line are recorded as an error on the line and dropped, because no group owns them and no crawler applies them.

When a crawler is looked up, each token in each group is compared against the crawler's own name, lowercased. A token must match from the start of the name, and the longest matching token wins. Groups that name the same token are combined into one, and a group for * is never combined with a specific one. * only counts when no specific token matched anything, which is why a site with a GPTBot group and a wildcard group gives GPTBot the GPTBot rules and nothing else. A crawler with no group of its own and no wildcard to fall back on is unrestricted, and the row says so rather than reporting a verdict pulled from another group.

Path matching turns each pattern into a regular expression

For the path tester, each Allow and Disallow pattern is converted to a regular expression and tested against the path you typed. A trailing $ anchors the match to the end of the URL and is stripped from the pattern. Every * becomes .*. Every other regular expression special character is escaped so it is matched literally, which matters because a path containing a ? or a + would otherwise be treated as a quantifier and match the wrong URLs entirely. The result is anchored at the start, and only at the end when the dollar sign was there, so a rule for /search matches /search?q=ip and /searching but a rule for /search$ matches only the exact path /search, not /search?q=ip or /searching.

All matching patterns are collected and the longest wins, because pattern length is how specificity is measured. When an Allow and a Disallow are the same length, the Allow wins. The winning rule is printed with its opposite beside it so you can see exactly which comparison decided the answer, rather than being handed a verdict with no reasoning attached.

The checks list is assembled from three severities

Issues are collected as they are found and never reordered until the very end. Errors are lines a crawler cannot use; warnings are lines that parse but probably do not do what was intended; notes are informational, such as a Yandex-only directive. When the list is printed it is sorted by severity so errors come first, capped at twenty-four entries, and anything beyond the cap is summarised as a count rather than silently dropped. A file larger than 500 KiB raises an error before any line is parsed, because only the first 500 KiB is read by a crawler and anything past that limit is ignored without warning. A body that looks like HTML — a leading <!doctype, <html or <?xml — also raises an error, because a file that is actually a web page has no directives in it, and the practical effect is the same as having no robots.txt at all.

When Would You Use This Tool?

What Does Each HTTP Status Do to Crawling?

The status code matters as much as the contents, because it largely decides whether the rules are used at all. The behavior below is as documented by Google Search Central (page last updated August 31, 2026, checked October 2, 2026) and follows RFC 9309 where that standard covers it. Other crawlers can differ.

How Does It Fit With DNS and Hosting?

A crawler cannot read your rules until it has resolved your domain and reached your server, so an unreachable file is often a DNS or hosting problem rather than a robots problem. If Search Console reports the file as unreachable, confirm the records with the DNS Lookup tool and check that the host answers with the Ping tool before you edit a single line.

Groups are matched against the crawler's user-agent token, not against its IP address. That is why the name in a log line proves nothing on its own: anything can claim to be Googlebot, and the User Agent Checker shows how easily that string is set. Verification usually means a reverse DNS lookup on the address that made the request, or a check against the IP ranges the crawler's operator publishes.

What Other Tools Work With This One?

Sources and Further Reading

• RFC 9309: Robots Exclusion Protocol. The standard for groups, Allow and Disallow, wildcards and status handling.
• How Google interprets the robots.txt specification: status code handling, caching, file size limit, group matching and rule precedence.
• Google's common crawlers: user-agent tokens, including what Google-Extended controls.
• Verify requests from Google: how to confirm that a request really came from a Google crawler.

Frequently Asked Questions