Does robots.txt keep a page out of search results?

Q: Does robots.txt keep a page out of search results?

Robots.txt controls crawler access, not secrecy or guaranteed exclusion from search. Google explains that a disallowed URL may still appear when discovered through other links. Choose indexing controls for indexing decisions, and authentication for private information. Review rules for the intended crawler rather than assuming one directive covers every consumer.

Separate crawling from indexing

A crawler uses robots.txt to decide which URLs it may request. A search engine can learn that a URL exists without fetching its full content. That is why blocking a path is not the same as guaranteeing it will disappear from results.

Protect private material properly

Do not put confidential documents on a public URL and rely on a robots rule to protect them. Use authentication and access controls. For public pages that should not be indexed, review the appropriate noindex mechanism and ensure the crawler can actually observe it; a blocked request may prevent that observation.

Use noindex on a public HTML page

For a public HTML page you want excluded from Google Search, place this tag inside that page’s <head>:

<meta name="googlebot" content="noindex">

Google must be able to crawl the page to read this tag: do not block that URL in robots.txt. Google does not support noindex in robots.txt. After publishing, check the page’s HTML for the tag and use Search Console’s URL Inspection to review what Google has crawled. A code change alone does not establish removal from results. See Google’s noindex guidance.

Check the deployed file

Inspect the live robots.txt, including the hostname, path, status and text content. A successful HTTP response that contains an HTML error page is not a useful robots file. Review crawler-specific groups and sitemap links with a person who understands the desired policy. The lab reports whether it observed a plain-text file; it does not prove every rule permits or prevents access.