What a user agent is
A user agent identifies the software making an HTTP request. It commonly appears in the User-Agent request header. Browsers, command-line tools, apps, monitoring systems, and crawlers can all send user-agent strings.
User-Agent: ExampleCrawler/2.1 (+https://example.com/crawler)A crawler string often contains a product token, version, and a documentation URL. Format varies between operators. The string is descriptive metadata, not a cryptographic identity.
How robots.txt uses user agents
A robots.txt group begins with one or more User-agent lines. The crawler compares its product token with the available groups and follows the most specific applicable rules under its implementation of the protocol.
User-agent: ExampleCrawler
Disallow: /private-preview/
User-agent: *
Disallow: /temporary/The wildcard group applies to crawlers that do not match a more specific group. Rules should use the product token documented by the crawler operator rather than an invented browser-style string.
Why user-agent strings can be spoofed
Any HTTP client can write almost any value into the header. An abusive bot can claim to be Googlebot, a browser, or an old crawler. User-agent text therefore helps classification but cannot authenticate the requester.
Do not allow access to private data because a request claims to come from a trusted crawler. Use authentication and authorisation for protected resources.
How to verify a crawler
- Find the crawler operator's official verification instructions.
- Check whether the operator publishes IP ranges, DNS rules, or signed requests.
- Compare the source address with those official records.
- Review request behaviour, frequency, target URLs, and response patterns.
- Keep evidence before applying a permanent network block.
Google recommends verifying Googlebot through published IP ranges or a forward-confirmed reverse DNS process. Other operators use different methods, and some provide no reliable verification method.
Reading user agents in logs
Log analysis is more useful when the user-agent field is combined with:
- source IP address;
- request timestamp and frequency;
- requested path and query string;
- HTTP method and response status;
- bytes transferred and response time;
- robots.txt fetches and rule compliance.
A single request rarely proves much. Patterns across time help separate normal crawling, broken automation, vulnerability scanning, scraping, and spoofed identities.
Managing crawler traffic
| Goal | Appropriate control |
|---|---|
| Request reduced crawling from a compliant bot | Robots.txt rules and operator-specific controls |
| Protect private resources | Authentication and authorisation |
| Stop abusive request rates | CDN, WAF, rate limit, or server rule |
| Remove a page from a search index | Supported index-control directive on a crawlable response |
| Confirm search-engine crawling | Verified logs plus operator verification method |
Official reference
See Google's current Googlebot documentation for its published user agents and verification methods. The general robots matching framework is defined in RFC 9309.