What Should a Robots.txt File Look Like?

If you've ever wondered what search engine crawlers see when they visit your website before anything else, the answer is your robots.txt file. It's a plain-text file sitting at the root of your domain that tells bots which pages they're allowed to crawl — and which ones to leave alone. Getting it right matters for SEO, site performance, and controlling how your content is indexed.

What Is a Robots.txt File and Where Does It Live?

A robots.txt file is a simple text document that follows the Robots Exclusion Protocol (REP) — an informal but widely respected standard that search engines like Google, Bing, and others follow voluntarily.

It must always live at the root of your domain, accessible at:

It cannot be placed in a subfolder and still function correctly. If a crawler can't find it at the root, it typically assumes no restrictions exist and crawls everything.

The Basic Structure of a Robots.txt File

A valid robots.txt file is made up of directives — simple instructions written in a specific format. Here's what a minimal, correctly structured file looks like:

Let's break down each component:

• User-agent — Specifies which bot the rules apply to. The asterisk (*) is a wildcard meaning "all bots." You can also target specific crawlers like Googlebot or Bingbot.
• Disallow — Tells the specified bot not to crawl the listed path. A blank Disallow: value means nothing is disallowed.
• Allow — Explicitly permits crawling of a path, even within a disallowed directory. Not all crawlers support this directive equally.
• Sitemap — A non-required but strongly recommended line that points crawlers to your XML sitemap.

Each group of rules (called a record) applies to one user-agent. A blank line separates records.

Common Robots.txt Patterns and What They Mean

One important point: robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it. To prevent indexing entirely, you need a noindex meta tag — but crawlers must be able to access the page to read that tag in the first place. This is a common source of confusion.

What a Well-Structured Robots.txt File Looks Like in Practice 🔍

A reasonably configured robots.txt for a typical website might look like this:

This example:

• Blocks all bots from admin, checkout, cart, and account pages
• Allows a specific WordPress file that needs to be accessible
• Completely blocks a specific AI crawler (GPTBot) from the entire site
• Points all crawlers to the sitemap

Variables That Determine What Your File Should Contain

There's no universal robots.txt that works for every site. What yours should look like depends on several factors:

Platform and CMS — WordPress, Shopify, and custom-built sites each generate different URL structures. A WordPress site may need to protect /wp-admin/, while a Shopify store might block /collections/filter URLs that create duplicate content.

Site purpose — An e-commerce site needs to think carefully about blocking cart and checkout pages. A news site might need to restrict archive URLs. A development site in staging should block everything.

Crawl budget — Larger sites with thousands of pages benefit more from robots.txt optimization. Blocking low-value pages (pagination, search result pages, filtered URLs) helps search engines spend their crawl budget on pages that actually matter.

Duplicate content concerns — If your CMS generates multiple URLs for the same content (with and without trailing slashes, session IDs, or URL parameters), robots.txt can help reduce what gets crawled — though canonical tags are often a better tool for this.

AI and data scraping — A growing number of webmasters are now adding specific blocks for AI training crawlers like GPTBot, Google-Extended, and CCBot. Whether to do this depends on your views on how your content is used.

What a Robots.txt File Should Never Do

A few things robots.txt is not designed for:

• 🚫 It does not replace authentication or access control. A Disallow rule doesn't prevent someone from directly visiting a URL — it only tells polite bots not to crawl it.
• It does not guarantee that a page won't be indexed — only that it won't be crawled by compliant bots.
• It does not affect internal site search or server-side rendering.
• Malicious bots routinely ignore robots.txt entirely.

Syntax Mistakes That Break Robots.txt

Even small errors can cause unintended consequences:

• Case sensitivity — Paths are case-sensitive on most servers. /Admin/ and /admin/ are treated as different paths.
• Missing trailing slash — Disallow: /private blocks that exact URL but may not block /private/page. Use /private/ to block the directory.
• Commenting too aggressively — Comments (lines starting with #) are fine, but some older parsers handle them inconsistently.
• Ordering of rules — When Allow and Disallow rules conflict, most crawlers apply the most specific rule that matches. Google uses the longest matching rule.

How Different Sites End Up with Very Different Files

A simple blog with a few dozen pages might have a robots.txt that's five lines long and blocks only the admin panel. A large e-commerce site might maintain dozens of Disallow entries targeting filtered product pages, internal search results, and user account areas. A site under development might block all crawlers entirely.

The same platform, different configurations, different content strategies, and different SEO goals all produce robots.txt files that look meaningfully different from one another — and all of them could be correct for their context.

What the right file looks like for any given site comes down to the specific pages that exist, how they're structured, and which ones genuinely deserve to be crawled and indexed.