Robots.txt: syntax, rules, and limits.

What robots.txt controls

A robots.txt file tells compliant crawlers which URL paths they may request from a specific protocol, host, and port. Its main use is crawl management. It does not make a public URL private and does not reliably remove a URL from a search index.

Correct location and format

Place the file at the root:

https://example.com/robots.txt

Use plain text and UTF-8-compatible content. Rules on https://example.com/robots.txt apply to that HTTPS host. They do not automatically control http://example.com, https://www.example.com, or another port.

Core directives

Google documents support for user-agent, allow, disallow, and sitemap. It does not support crawl-delay. Other crawlers may implement additional fields, so check their documentation before relying on one.

Basic examples

Allow normal crawling

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Block one directory from all compliant crawlers

User-agent: *
Disallow: /temporary/

Allow one path inside a blocked directory

User-agent: *
Disallow: /documents/
Allow: /documents/public/

Use crawler-specific rules

User-agent: ExampleBot
Disallow: /

User-agent: *
Disallow: /staging-preview/

Path matching

Rules match URL paths. Under RFC 9309, crawlers should use the most specific matching rule. Percent-encoding and character handling can affect matches, so test important patterns with the crawler operator's tools or parser.

Some major crawlers support pattern characters such as * and an end anchor such as $, but extensions are not implemented identically everywhere. Prefer simple path prefixes when they meet the requirement.

Robots.txt versus noindex

Blocking crawling prevents a compliant crawler from fetching the page. A crawler that cannot fetch a page also cannot read a robots meta tag placed in that page. For search removal, the usual pattern is to allow crawling and return a supported noindex directive until the search engine processes it.

How responses affect robots.txt

Crawler behaviour can differ when robots.txt returns a success, redirect, not-found, or server-error response. RFC 9309 defines general availability handling, while search engines publish their own operational details. Monitor the file as a production endpoint and avoid accidental 5xx failures.

Common implementation errors

• Uploading the file to a subdirectory rather than the host root.
• Blocking CSS or JavaScript required to render important content.
• Blocking pages that need to be crawled so a noindex directive can be seen.
• Using a robots rule as protection for confidential resources.
• Copying platform rules without checking the site's actual URL structure.
• Using stale sitemap URLs.
• Assuming every crawler supports crawl-delay or the same wildcard syntax.
• Deploying a sitewide Disallow: / from a staging environment.

Testing checklist

1. Open the exact robots.txt URL in a browser.
2. Confirm a 200 response and plain-text content.
3. Test representative allowed and blocked URLs.
4. Check separate hostnames and protocols independently.
5. Review server logs to see whether target crawlers fetch the file.
6. Recheck rules after migrations, redesigns, and CMS changes.
7. Confirm that sitemap declarations use absolute, current URLs.

Conservative production example

User-agent: *
Disallow: /internal-search/
Disallow: /preview/
Allow: /preview/public-example/

Sitemap: https://example.com/sitemap.xml

This example manages crawl paths but does not secure them. Any sensitive directory still needs real access control.

Official sources

Use RFC 9309 for the standard protocol and Google's current robots.txt introduction and parser documentation for Google-specific behaviour.