How to Create a Robots.txt File: A Complete Guide for Web Developers

Every website that wants search engines to crawl it — or not crawl certain parts of it — needs a robots.txt file. It's one of the simplest files in web development, yet it's frequently misunderstood, misconfigured, or skipped entirely. Here's what it actually is, how it works, and what goes into building one correctly.

What Is a Robots.txt File?

A robots.txt file is a plain-text file placed at the root of your website that tells web crawlers (like Googlebot, Bingbot, or any other automated spider) which pages or sections of your site they're allowed to access. It follows the Robots Exclusion Protocol, a standard that's been in place since 1994.

When a crawler visits your site, the first thing it typically checks is https://yourdomain.com/robots.txt. Based on the instructions it finds there, it either proceeds to crawl specific content or skips it.

⚠️ Important distinction: robots.txt controls crawl access, not indexing. A page blocked in robots.txt can still appear in search results if another site links to it. To prevent indexing, you need a noindex meta tag or HTTP header — robots.txt alone won't do that job.

How to Create a Robots.txt File Step by Step

Step 1: Open a Plain Text Editor

Create a new file using any plain-text editor — Notepad on Windows, TextEdit on Mac (in plain-text mode), or a code editor like VS Code. Do not use a word processor like Microsoft Word. The file must contain no formatting, no special characters, and no hidden markup.

Step 2: Understand the Basic Syntax

A robots.txt file is built from a small set of directives:

A user-agent of * applies to all crawlers. You can also target specific bots by name (e.g., Googlebot, Bingbot).

Step 3: Write Your Rules

Here's what a basic, functional robots.txt file looks like:

This tells all crawlers: don't access /admin/ or /private/, but everything else is fair game. The Sitemap line helps crawlers find your structured site map.

To block all crawlers from the entire site (useful during development):

To allow all crawlers full access with no restrictions:

Leaving Disallow empty signals unrestricted access — it's not the same as omitting the file entirely, but the effect is similar.

Step 4: Save the File Correctly

Save the file as robots.txt — lowercase, with no other extension. The filename must be exact. Upload it to the root directory of your website, so it's accessible at yourdomain.com/robots.txt. If it lives anywhere else, crawlers won't find it.

Step 5: Test Your File 🔍

Before treating it as live and final, test it using Google Search Console's robots.txt Tester (found under the legacy tools section). This tool shows you how Googlebot interprets each rule and flags syntax errors. You can also manually visit yourdomain.com/robots.txt in a browser to confirm the file is publicly accessible.

Variables That Affect How You Configure Robots.txt

There's no single correct robots.txt file — what you include depends heavily on your situation.

CMS platform: WordPress sites often need to disallow /wp-admin/ but allow /wp-admin/admin-ajax.php (which some plugins rely on). Shopify and other hosted platforms sometimes auto-generate robots.txt files with limited or no customization options.

Site architecture: A large e-commerce site with thousands of filtered product URLs (like /products?color=red&size=M) may need to disallow parameter-based URLs to prevent crawl budget waste. A small brochure site rarely needs that level of nuance.

Technical SEO goals: If you're trying to conserve crawl budget — the number of pages Googlebot will process on a given crawl — blocking low-value URLs (duplicate content, internal search results, session IDs) becomes important. For a new or small site, crawl budget is rarely a concern.

Staging environments: A development or staging site should block all crawlers entirely to prevent duplicate content issues and accidental indexing of unfinished pages.

Third-party tools: Some services (ad networks, analytics crawlers, monitoring bots) use their own user-agent strings. You can write targeted rules for specific bots without affecting how search engine crawlers see your site.

Common Mistakes That Break Robots.txt

Blocking CSS and JavaScript files is one of the most damaging errors. If Googlebot can't render your page because its supporting files are blocked, it may misread the content and hurt your rankings. Google's guidelines explicitly recommend not blocking these resource files.

Using wildcards incorrectly is another frequent issue. The * wildcard in Disallow paths works differently than you might expect — it matches any sequence of characters, but not all crawlers implement it identically. Google supports * and $ (end-of-URL anchor); simpler crawlers may not.

Assuming robots.txt is security: It isn't. Any crawler — or person — can ignore it. Sensitive pages need proper authentication, not just robots.txt exclusion.

The Part That Depends on Your Setup

The mechanics of robots.txt are straightforward. The configuration decisions are not. How aggressively you disallow paths, which bots you target individually, whether you need crawl-delay rules, and how your file interacts with your sitemap and meta tags — those answers shift significantly depending on your site's size, platform, indexing goals, and how much of your content is genuinely unique versus duplicated or auto-generated.

A small personal blog and a large multi-language e-commerce platform both use the same file format, but what goes inside that file looks nothing alike.