Yes. A URL blocked by robots.txt can still appear in search results. This surprises many site owners because the word “block” sounds absolute. In practice, robots.txt asks compliant crawlers not to request selected URLs. It does not erase knowledge of those URLs, and it does not provide a dependable noindex command.
The distinction matters whenever you handle private areas, duplicate pages, faceted navigation, staging sites, PDFs, or old content. Using the wrong control can leave a URL visible with a bare title or link-based description even though the crawler never fetched the page itself.
What robots.txt actually controls
The Robots Exclusion Protocol is a standardised method for publishing crawl preferences at the root of a site. The current technical standard is RFC 9309. A robots.txt file defines rules for user-agent groups and URL path patterns. Compliant crawlers evaluate those rules before requesting a resource.
A Disallow rule therefore controls access to the resource, not whether the URL can exist in a search engine’s records. Search engines may learn the address through internal links, external links, redirects, sitemap files, old crawl data, browser data, or other discovery systems.
How a blocked URL can enter search results
Imagine that many public pages link to /private-report.pdf, but robots.txt blocks that path. The crawler can see the links and anchor text on those public pages. It knows the target URL exists, yet it cannot fetch the PDF to inspect its contents or read an indexing directive.
The search engine may keep the URL as a known resource and show it for a narrow query. The result can have limited information because the engine could not crawl the page. Google notes that blocked pages may still appear, although usually without a useful description.
This is why a blocked result sometimes shows wording based on surrounding links rather than the page’s own title and text. The search engine is working from indirect signals.
Why noindex cannot work behind a crawl block
A robots meta tag and an X-Robots-Tag header are delivered with the page response. A crawler must request the URL to read them. If robots.txt prevents that request, the crawler cannot see the noindex instruction.
Google’s guide to blocking indexing with noindex states that the resource must remain accessible to the crawler for the rule to take effect. This is one of the most common technical SEO conflicts: the site blocks crawling and adds noindex, expecting two controls to reinforce each other, when the crawl block can stop the indexing rule from being processed.
Choose the control based on your real objective
Keep a public page out of search
Allow the crawler to access the page and return a noindex directive. Use a robots meta tag for HTML or an X-Robots-Tag header for non-HTML files. Keep the page out of XML sitemaps and avoid unnecessary internal links while the change is processed.
Protect confidential information
Use authentication or server-side access control. Neither robots.txt nor noindex provides security. Robots.txt is publicly readable, and noindex still allows anyone with the URL to load the resource.
Reduce crawling of low-value URL spaces
Robots.txt can be suitable for infinite calendars, internal search results, and certain parameter combinations when those URLs do not need to be crawled. However, first make sure that blocking them will not prevent search engines from reaching important content or reading needed canonical signals.
Remove an old page permanently
Return 404 or 410 when the content is genuinely gone and has no replacement. Use a permanent redirect when a clear replacement exists. Search engine removal tools can hide a result temporarily, but the underlying technical state still needs to be correct.
A realistic removal sequence
Suppose an outdated public page is indexed and you want it removed while keeping it available to direct visitors. First remove any Disallow rule that blocks the URL. Add noindex to the HTML or response header. Confirm that the page still returns 200 and that the directive is visible without authentication.
Then request recrawling through the relevant search engine tools and wait for processing. Once the URL is no longer indexed, you may decide whether a crawl restriction is still necessary. In many cases it is not, because a correctly noindexed page can remain crawlable with minimal practical cost.
Before making changes, follow the checks in how to test page crawlability. That prevents a hidden redirect, header rule, or server error from complicating the removal.
Common robots.txt mistakes
- Using Disallow as if it were a security control.
- Blocking a page before the crawler can read its noindex directive.
- Listing non-canonical URLs in a sitemap while blocking them from crawling.
- Applying a rule to the wrong hostname.
- Blocking CSS or JavaScript needed to render important pages.
- Copying broad wildcard rules without testing the affected paths.
The rule to remember
Robots.txt answers a narrow question: may this crawler request this URL? Indexing controls answer a different question: may this resource appear in search results? Access control answers a third question: may this visitor retrieve the resource at all?
Choose the mechanism that matches the objective. If the goal is privacy, require authentication. If the goal is deindexing, use noindex while keeping the URL crawlable. If the goal is managing unnecessary crawling, robots.txt may be appropriate. Treating these controls as interchangeable creates gaps that are easy to avoid once their roles are separated.