A robots.txt file is a plain text file at the root of a website that tells web crawlers which parts of the site they're allowed to request. Reputable crawlers check it before requesting anything else, which makes it the first line of control over who accesses what - though it's a request, not a lock, so it only works with crawlers that choose to respect it.
Every site with visitors also has crawlers - search engines indexing it, AI systems reading it, tools checking it for one reason or another - and most of them show up unannounced. Robots.txt is the one file that lets a site owner say something back before any of that traffic touches a page.
This guide covers what the file actually does, what it can't do, a real one you can read in full, and the handful of mistakes that turn a helpful file into a self-inflicted problem.
What Is a Robots.txt File?
It's a plain text file, always named robots.txt and always placed at a site's
root - example.com/robots.txt, never in a folder. It lists rules by crawler,
telling each one which paths it may or may not request. Nothing about it requires special
software to read or write; it's a file format simple enough to write by hand in a few
minutes, which is part of why it's stayed unchanged for decades.
How Does a Robots.txt File Actually Work?
A well-behaved crawler requests /robots.txt first, before it requests anything
else on the site. It reads the rules that apply to its own name (or to *, meaning
everyone), and skips whatever those rules disallow. That name is the same one a crawler sends
in its User-Agent header on every request - MyIPShow's own
User Agent Check tool
shows what that string looks like from your own browser, which is the same field a
User-agent: line in robots.txt is matching against.
One file, checked once, decides the crawler's whole visit path.
| Directive | What It Does | Example |
|---|---|---|
| User-agent | Which crawler the rules below apply to | User-agent: * |
| Disallow | Blocks that crawler from a path | Disallow: /search |
| Allow | Explicitly permits a path, overriding a broader Disallow | Allow: / |
| Sitemap | Points crawlers to the site's sitemap file | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Asks for a pause between requests - not honored by Google | Crawl-delay: 10 |
Why Does a Website Need One?
Without one, every crawler assumes it can request everything, which is rarely what a site actually wants. Search results pages, admin logins, duplicate parameter URLs and internal tools all get crawled the same as the content meant for readers, which wastes crawl budget on pages nobody needed indexed in the first place. A robots.txt file is how a site says which parts are worth a crawler's time and which aren't - a small file doing a job that otherwise falls to guesswork.
What Does MyIPShow's Own Robots.txt Look Like?
Rather than a made-up example, here's this site's actual file. It has two groups. The first applies to every crawler:
User-agent: *Disallow: /search- Blogger's internal search-results pages, kept out of crawling since they're duplicates of content that already has a real URL.Allow: /- everything else is open.
The second names a set of AI crawlers specifically - GPTBot, ChatGPT-User, OAI-SearchBot,
PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended,
GoogleOther, Bingbot and CCBot - and gives them the exact same two rules. None of them are
blocked; the group exists to be explicit rather than to rely on the wildcard covering them by
default. The file closes with two Sitemap lines, pointing to Blogger's own auto-generated
/sitemap.xml and /sitemap-pages.xml feeds.
One thing that didn't make it in: a Crawl-delay line. Blogger's own robots.txt validator rejects a crawler group that has a Crawl-delay with no Allow or Disallow alongside it, and Google doesn't honor the directive anyway - one more reason it's absent rather than an oversight.
Does Robots.txt Block AI Chatbots From Using Your Content?
It can ask them to stay out, and the well-known ones publish a name specifically so a site can address them. OpenAI documents separate agents for training versus live use - OpenAI's overview of its crawlers lists GPTBot for training data collection and OAI-SearchBot for its search feature as independent settings, so blocking one doesn't automatically block the other. Google runs the same split: Google-Extended controls AI training use separately from Googlebot's regular search crawling.
The part worth being clear-eyed about: robots.txt is a request a crawler chooses to honor, not a lock. Reputable companies publish their crawler names and generally respect the file. Nothing stops a less scrupulous scraper from ignoring it entirely and using a name that isn't even in the file.
What Common Robots.txt Mistakes Break a Site?
- Disallowing the entire site by accident.
Disallow: /underUser-agent: *blocks everything, not just the one folder someone meant to hide. - Forgetting the Sitemap line. It's optional but costs nothing to include, and it gives crawlers a direct path to every URL worth indexing.
- Blocking the CSS or JS a page needs to render. A crawler that can't load a page's styling can misjudge what the page actually looks like.
- Putting the file anywhere but the root. A robots.txt inside a subfolder is invisible to crawlers - it only counts at the domain's top level, and a copy on one subdomain doesn't cover another. If a site spans multiple subdomains, a DNS Lookup confirms which subdomain a given address actually resolves to before you assume one robots.txt file is covering all of them.
Does Robots.txt Affect Google Rankings?
Not directly, and it's not built to keep a page out of Google Search at all. Google's own documentation is explicit about this: Google's introduction to robots.txt states plainly that a disallowed page can still be indexed and shown in search results if other pages link to it - it just won't have a description, because Google never crawled it to write one. Keeping a page fully out of search results needs a noindex tag or password protection, not a Disallow rule. What robots.txt does affect indirectly is crawl efficiency: keeping crawlers off low-value pages leaves more of their attention for the pages that matter.
How Do You Check or Test a Robots.txt File?
The most direct check is also the simplest: type any site's address followed by
/robots.txt into a browser and read the raw file. For a faster read on what the
rules actually mean for a specific crawler and path, the
Robots.txt Analyzer tool
parses the file and shows what's allowed and what isn't, without requiring you to trace the
directives by hand.
Frequently Asked Questions
Is a robots.txt file required for every website?
No. Without one, crawlers assume they can request everything, which works fine for a small site with nothing to hide from crawling. It becomes useful once a site has search-results pages, admin areas or duplicate URLs it would rather crawlers skipped.
Does Disallow in robots.txt remove a page from Google?
Not by itself. Google's own documentation states that a disallowed page can still appear in search results without a description if other pages link to it. Removing a page from search results entirely needs a noindex tag or password protection instead.
Can I block ChatGPT or other AI bots with robots.txt?
Yes, for the crawlers that publish a name and respect the file - OpenAI documents GPTBot and OAI-SearchBot as separate, independently controllable agents, for example. It only works against crawlers that choose to honor robots.txt in the first place.
What happens if a robots.txt file has no rules at all?
An empty or missing robots.txt is treated the same way: crawlers assume everything is allowed. There's no need for a file that says "allow everything" explicitly, since that's the default.
Does Google support the Crawl-delay directive?
No. Google's documented, supported fields are User-agent, Allow, Disallow and Sitemap - Crawl-delay isn't one of them, even though some other crawlers do honor it.
Where does a robots.txt file need to be placed?
At the root of the domain - example.com/robots.txt - and nowhere else. A copy placed inside a subfolder is invisible to crawlers, since they only ever check the root location.
Final words. A robots.txt file is a short, plain-text request that most crawlers genuinely respect - useful for steering them away from pages that don't need indexing, but never a substitute for noindex or a password when the goal is actually keeping something out of search. Read any site's own file directly, or check what a set of rules actually permits with the Robots.txt Analyzer tool.
