Questions about crawlers, bots, and indexing.

A web crawler is software that automatically requests pages and other files, follows links, and records information for a service such as a search engine, archive, monitoring platform, or research system.

No. Crawling is the process of fetching a URL. Indexing is the later process of analysing a fetched resource and deciding whether and how it can be stored in a search index. A page can be crawled without being indexed.

No. A sitemap helps a search engine find preferred URLs and associated metadata. It remains a discovery signal, not an instruction to index every submitted URL.

Not reliably. Robots.txt controls crawling, not indexing. A blocked URL may still be known through links and appear without a normal snippet. Use an index-control method such as a robots meta tag or X-Robots-Tag on a crawlable response when removal from an index is the goal.

It belongs at the root of the relevant protocol, host, and port, such as https://example.com/robots.txt. A file placed in a subdirectory does not control the whole host.

A user agent is an identifier sent in an HTTP request. Crawlers commonly use it to state which software made the request. The text can be copied or spoofed, so it is not proof of identity by itself.

Use the verification method published by the crawler operator. For Google crawlers, that can mean matching published IP ranges or completing a forward-confirmed reverse DNS check. Do not trust only the user-agent string.

Yes. Robots.txt is a protocol followed voluntarily by compliant crawlers. It is not an authentication or access-control system. Protect private or sensitive resources with proper authorisation.

Crawl budget is a practical description of how much crawling a search engine is willing and able to perform on a site. It matters most on large or frequently changing sites. Small sites usually gain more from fixing duplicate paths, server errors, weak internal linking, and accidental blocks.

Possible reasons include low perceived value, duplicate URL patterns, weak internal links, limited server capacity, a very large URL inventory, or scheduling delays. The status alone does not identify one guaranteed cause.

The page may be duplicate, thin, soft-404-like, canonicalised elsewhere, blocked by an index directive, or judged unsuitable for the index. Check the rendered content, response headers, canonical signals, internal links, and Search Console details.

Include canonical URLs that you want search engines to crawl and consider for indexing. Exclude redirects, error pages, duplicates, parameter variants, and URLs intentionally marked noindex.

The lastmod value is sitemap metadata about a meaningful page update. It should be accurate. It is not a ranking boost, and changing it without changing the page reduces its usefulness.

Major search engines can render many JavaScript pages, but rendering adds complexity and delay. Important content, links, status codes, canonical tags, and index directives should remain easy for crawlers to access and interpret.

Separate technical access problems from ranking or demand changes. Check server availability, robots.txt, index directives, canonicals, redirects, sitemap changes, manual actions, security issues, and whether the affected pages still return useful content.