Read any website's robots.txt file and see what it really says: which paths each crawler may request, which sitemaps are declared, and where the syntax is wrong. Results are parsed in your browser against the rules in RFC 9309.
Only the file at the root of the host is read, and always over https, for example https://example.com/robots.txt. A site is not obliged to let other pages read its files, so some hosts cannot be read from a browser. When that happens you can open the file yourself and use the Paste a file tab.
A robots.txt analyzer reads the file at the root of a site and shows what each crawler may request. Paste a domain or a robots.txt file to see every user-agent group, its Allow and Disallow rules, the sitemaps declared, and any syntax mistake a crawler would trip on. The file follows the Robots Exclusion Protocol (RFC 9309), and one wrong line can hide a whole section of a site from search.
What Is a Robots.txt Analyzer and What Does It Do?
What does this tool do?
It fetches the robots.txt file for a host, then parses it the way a crawler does. It splits the file into user-agent groups, counts the groups and the rules they hold, lists every sitemap, and reports the HTTP status the server returned, because that status alone decides whether the rules apply at all. It then tells you, for six well known crawlers, whether the site is fully open, partly restricted or blocked outright, and it flags syntax mistakes such as a path with no leading slash or a directive that search engines no longer support. When no usable file comes back, the crawler rows say so instead of guessing.
Why is this useful?
One misplaced line can hide an entire section of a site from search. A rule that blocks a folder of stylesheets or scripts stops Google rendering the page it is trying to index, and a stray Disallow: / left over from a staging server removes a whole site from crawling. Because robots.txt is fetched from a fixed location and cached for up to a day, mistakes are slow to surface and slow to undo, so it is worth reading the file as a machine reads it rather than as you remember writing it.
Who uses this tool?
• SEO specialists confirming that money pages, images and rendering assets are all crawlable.
• Web developers checking crawl rules before a redesign, a CMS migration or a domain move goes live.
• Site administrators keeping admin paths, search result pages and staging copies out of the index.
• Agencies auditing a client site they have no server access to.
• Publishers deciding which AI crawlers, such as GPTBot or Google-Extended, may read their content.
How Do You Use This Tool?
1. Type a domain such as https://example.com into the Website address box. Any path you include is ignored, because robots.txt only ever lives at the root, and the site you are reading now is checked automatically when the page loads.
2. Press Analyze, or the Enter key. The tool requests /robots.txt from that host over https and reports the HTTP status it got back.
3. Read the summary, then the per crawler verdicts, then the checks list. Errors are shown before warnings and notes.
4. Enter a path such as /admin/settings.html in the path tester and switch crawler tabs to see which rule decides it. Enter works there too.
5. If a host cannot be read from a browser, open its file in a new tab, copy the text into the Paste a file tab, and analyze it there.
Example: how one file answers three different requests
User-agent: * Disallow: /search Disallow: /admin/ Allow: /admin/help/ Sitemap: https://www.myipshow.com/sitemap.xml
/admin/help/start.html is allowed. Two rules match it, and the longer pattern /admin/help/ beats /admin/.
/admin/keys.html is disallowed. Only /admin/ matches, and nothing overrides it.
/search?q=ip is disallowed. Rules are prefix matches, so /search covers every URL that starts with those seven characters, query strings included.
How a crawler decides, in the order it decides it
A crawler obeys exactly one group. Rules in the other groups, including the group for *, are ignored once a more specific name matches.
How Do You Read the Results?
Fetch status
The HTTP status the server returned for /robots.txt, and what it means for crawling. A 2xx status means the rules are read and applied. Any 4xx other than 429 is treated as though the file did not exist, so crawling is unrestricted. A 429 or a 5xx is treated as a temporary full block. On anything that is not a 2xx there is no file to parse, so the sections below report the consequence of the status itself rather than a set of rules.
Groups and rules
A group is one or more consecutive User-agent lines plus the Allow and Disallow lines that follow them. The count tells you how much of the file is actually in force, since rules that sit above the first User-agent line belong to no group and are ignored.
Access for common crawlers
For each named crawler the tool resolves which single group applies, then evaluates the site root against that group. There are four outcomes. Full access means the group that applies holds no Disallow rule at all. Partly restricted counts the Disallow rules in that group, which is a count of rules and not of URLs. Whole site disallowed means a rule blocks / and nothing narrower overrides it. No matching group means the file names neither that crawler nor *, which leaves it unrestricted. When the fetch returned no file, every row says so instead of showing a verdict.
Sitemaps declared
Every Sitemap line found. These are independent of the groups, so they apply to all crawlers, and they must be absolute URLs. The line is optional, and a missing sitemap is not an error. If no file was returned this section says that, rather than reporting none.
Syntax and best practice checks
Errors are lines a crawler cannot use, such as a rule with no colon or a path written as a full URL. Warnings are lines that parse but probably do not do what you intended. Notes are informational, for example a directive that only some search engines read. At most twenty-four are listed, followed by a count of anything beyond that.
Test a single path
Enter a path and the tool shows the verdict for the selected crawler plus the exact rule that produced it. The tabs cover the same six crawlers as the access list above, so a path can be tested for each of them. On the Google-Extended tab, allowed means Google may use that path in its generative models, since that token is a permission control rather than a crawler. This is the fastest way to settle an argument about whether a URL is blocked, because it names the winning pattern instead of guessing.
What this tool cannot tell you
• It cannot prove a page is indexed or not indexed. robots.txt controls crawling; indexing is decided separately, and a blocked URL can still be listed if other sites link to it.
• It is not a security check. The file is public, it only asks politely, and a scraper that ignores it will not be stopped by it.
• It cannot read a host that refuses cross origin requests from a browser. In that case the file is unreadable here, not missing, and the Paste a file tab is the way round it.
• It reports the file for one host and one scheme only. https://example.com/robots.txt says nothing about blog.example.com or about the plain http version of the same site, and this tool always requests the https version.
When Would You Use This Tool?
Before a launch
Staging servers usually carry Disallow: /. Check it is gone from the live host before you announce the site.
After a migration
Confirm the new host serves a file at all, and that the Sitemap line points at the new domain rather than the old one.
Search Console warnings
When a URL is reported as blocked by robots.txt, paste the path into the tester to see which rule is responsible.
Rendering problems
If a page looks unstyled in a rendering test, check whether a broad rule is blocking the folder holding your CSS, JavaScript or images.
AI crawler policy
Verify that GPTBot, ClaudeBot, PerplexityBot and Google-Extended have exactly the access you decided to give them, and no more.
Auditing a site you do not own
The file is public, so you can see which sections a site keeps out of search without any access to its server.
What Does Each HTTP Status Do to Crawling?
The status code matters as much as the contents, because it decides whether the rules are used at all. The behavior below is documented by Google Search Central and follows RFC 9309.
| Condition | What Google's crawler does |
|---|---|
| 200, plain text body | Parses the file and applies the rules exactly as written. |
| 200, but the body is an HTML page | No valid directives are found, so the practical effect matches an empty file: nothing is restricted. |
| 3xx redirect | Follows at least five hops. Beyond that the file is treated as missing. |
| 4xx other than 429, including 401 and 403 | Treated as though no robots.txt exists, so the whole site may be crawled. |
| 429, or any 5xx | Treated as a temporary full disallow. After about 30 days of failure Google falls back to its last cached copy, or assumes no restrictions if it has none. |
| File larger than 500 KiB | Only the first 500 KiB is parsed. Anything after that is ignored. |
| crawl-delay, noindex or host lines | Ignored by Google. Crawl-delay is read by Bing and Yandex; noindex has not worked in robots.txt since 2019. |
How Does It Fit With DNS and Hosting?
A crawler cannot read your rules until it has resolved your domain and reached your server, so an unreachable file is often a DNS or hosting problem rather than a robots problem. If Search Console reports the file as unreachable, confirm the records with the DNS Lookup tool and check that the host answers with the Ping tool before you edit a single line.
Groups are matched against the crawler's user-agent token, never against its IP address. That is why the name in a log line proves nothing on its own: anything can claim to be Googlebot, and the User Agent Checker shows how easily that string is set. Real verification means a reverse DNS lookup on the address that made the request.
About the data
The tool first asks for the file directly from your browser, always over https. That works for hosts that permit cross origin reads, and it is the fastest and most private path. When a host refuses, the request is retried once through the public AllOrigins relay, which returns the file together with the original HTTP status code; the domain you typed is visible to that service. Pasted files are never sent anywhere, and a pasted file is not added to Recent checks, because there is no host to check again. Recent checks are kept in your browser's local storage and are cleared with the Clear button.
What Other Tools Work With This One?
Frequently Asked Questions
What does the Disallow directive actually do?+
It asks a crawler not to request URLs that start with the given path. It is a request, not a block: search engines that follow the protocol obey it, while scrapers that ignore the protocol will fetch the URL anyway. Nothing about the page changes, and the file itself does not restrict human visitors.
Does blocking a page in robots.txt remove it from Google Search?+
No. Disallowing a URL stops Google fetching the content, but the URL can still be listed if other pages link to it, usually with no description. To keep a page out of the results, let crawlers reach it and use a noindex meta tag or an X-Robots-Tag header instead. A page blocked by robots.txt can never be read, so its noindex tag is never seen.
Is robots.txt enough to protect private pages?+
No, and it often makes things worse. The file is public at a fixed address, so every path you list is an invitation to look. Anything genuinely private needs authentication or a server side access rule, and should not appear in robots.txt at all.
Why would robots.txt block CSS or JavaScript files?+
Almost always by accident, through a broad rule such as a disallowed assets, includes or scripts folder. Google renders pages before indexing them, so when the stylesheet and script files are unreachable it sees a broken layout and may judge the page differently. Blocking a whole directory is convenient and rarely what you meant.
How do the wildcard and the dollar sign work in a path?+
An asterisk stands for any run of characters, so a rule for /*.pdf covers every path that ends with that extension anywhere on the site. A dollar sign anchors the end of the URL, so /*.pdf$ matches a plain PDF path but not the same path with a query string added. Anywhere else in a pattern both characters are treated literally. Both are part of RFC 9309 and are supported by Google, Bing and Yandex.
Does one robots.txt file cover subdomains and both http and https?+
No. A file applies to exactly one origin, meaning one scheme, one host and one port. The file on https://example.com governs that address only. A subdomain such as blog.example.com needs its own file, and strictly so does the plain http version of the same site. This is the single most common surprise when a site is served from more than one hostname.
Do AI crawlers obey robots.txt?+
The major named ones state that they do, and each publishes a token you can target: GPTBot for OpenAI, ClaudeBot for Anthropic and PerplexityBot for Perplexity. Google-Extended is not a crawler at all but a control token for whether Google may use your content in its generative models, and it does not affect Search. Unnamed scrapers are a different matter, and only a server side rule will stop those.
Why can this tool sometimes not read another site's file?+
A browser only lets a page read a response from another domain when that domain opts in with a cross origin header, and very few sites set one for robots.txt. When the direct read fails the tool retries once through a public relay, and if that is unavailable as well it says so rather than showing invented content. Opening the file in a new tab and pasting it into the Paste a file tab always works.