Robots.txt and noindex: control crawling and indexing safely

Three different controls

Robots.txt asks cooperating crawlers not to fetch selected URLs. Noindex asks a search engine to leave a fetched resource out of its index. Authentication controls who can actually access private content. Neither robots.txt nor noindex is a security boundary.

Why a block can be right—or costly

A public service page accidentally marked noindex can disappear from search, while a staging site may correctly need restrictions. A disallowed URL can still be indexed without its content. Blocking crawling can prevent Google from seeing a noindex directive on that page.

Decide what this URL should do

For a public page you want in search, allow crawling and remove unintended noindex directives. For a public page you want excluded, allow the crawler to retrieve its noindex directive. For private information, require authorization instead of relying on search-engine instructions.

Do not remove all restrictions sitewide to fix one warning. First identify which environment, path and crawler the rule is meant to cover, and who owns it.

1. Inspect HTML and response headers

Check meta robots and crawler-specific tags, plus the X-Robots-Tag HTTP header. An HTTP directive can affect PDFs and other non-HTML files too. Review combined instructions rather than assuming an index tag cancels noindex. Nofollow is a separate link-following instruction; it is not a substitute for noindex.

The robots-special-directives check reports nosnippet and noarchive when present. Nosnippet actively prevents text snippets and video previews in Google Search; keep it only if that restriction is intentional. Noarchive historically controlled cached links, but Google Search no longer uses it because cached links have been removed. Treat noarchive as a historical diagnostic, not an active Google restriction; check other crawlers’ documentation before removing it.

2. Read the matching robots.txt group

Fetch /robots.txt from the exact protocol and host being audited. A broad Disallow: / deserves immediate review on a public site. Consider user-agent groups and path specificity: rule order alone is not a reliable way to resolve apparent conflicts. If fetching failed, an informational audit result is not proof that the URL is allowed.

3. Check sitemap declarations

Confirm each Sitemap URL is intentional and reachable. A sitemap hosted elsewhere can be legitimate, so a different hostname needs investigation rather than automatic replacement. Sitemap declarations help discovery; they do not override noindex or a crawl restriction.

4. Test the intended outcome

After deployment, repeat the HTTP and HTML checks on affected URLs and an unaffected control page. For an indexing issue, inspect the URL in Search Console after Google has had time to process the change. Keep staging access protections in place while testing production.

Example: exclude a public utility page

Place this in the HTML head only when the page should remain publicly accessible but out of search. Do not block the same URL in robots.txt if Google needs to read this directive.

<meta name="robots" content="noindex">

Next.js: keep private areas separate

Set robots metadata on the appropriate server page or private-area layout, not the public root layout. This controls search instructions only: authenticated routes still need server-side authorization.

export const metadata = {
  robots: { index: false, follow: false },
};