X-Robots-Tag

What it is for

A robots meta tag only works on HTML pages, because it has to sit inside a <head> element. A PDF, an image file or a video has no head to put a tag in. X-Robots-Tag solves that by sending the same directive as an HTTP response header instead, which works on any file type a web server can respond to.

Google's own documentation gives this exact use case: adding a noindex, nofollow header to every PDF on a site through the server configuration, rather than editing each file.

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

The same pattern works for images, where putting a robots meta tag inside the file is not possible at all:

<Files ~ "\.(png|jpe?g|gif)$">
  Header set X-Robots-Tag "noindex"
</Files>

How it works

X-Robots-Tag and the robots meta tag are two implementations of the same rule set, noindex, nofollow, nosnippet, max-snippet and the rest, chosen by content type and by how much control a publisher has over the server. A meta tag needs edit access to the page's HTML; an X-Robots-Tag rule set in the server configuration applies to every matching file across the whole site at once, including files a CMS does not let an editor touch directly.

Because it is a header, not markup, a crawler only ever sees it by requesting the resource. That dependency matters: Google's own documentation on blocking indexing states plainly that for a noindex rule to work, whether set as a meta tag or as an X-Robots-Tag header, "the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler." A resource disallowed in robots.txt is never fetched, so the header is never read, and the resource can still turn up in search results through another path, such as a link from a page the crawler could reach.

Why it matters for AI visibility

AI Overviews, AI Mode and the crawl-based features other AI assistants build on inherit the same indexing signals Google Search uses, including noindex, nosnippet and max-snippet. A PDF whitepaper, a product spec sheet or a set of images are common sources a brand would want an AI system to draw on, or deliberately keep out of a generated answer, and a meta tag cannot reach any of them. X-Robots-Tag is the only way to apply that control at scale to non-HTML content, which is exactly the content type GEO work tends to overlook because it sits outside the normal page-by-page SEO checklist.

Good practices

• Use X-Robots-Tag for any file type that cannot carry a <meta> tag: PDFs, images, video files and other non-HTML resources.
• Match the server rule to a specific file extension or path, not a blanket rule that could catch files meant to stay indexable.
• Confirm the header actually reaches the response with a direct request, curl -I, rather than assuming the server config applied correctly.
• Check that the resource is not simultaneously disallowed in robots.txt, since a disallowed resource never lets the crawler see the header at all.
• Document which server config file owns the rule, since .htaccess, httpd.conf and CDN-level rules can each set conflicting headers for the same path.

Common mistakes

• Setting noindex via X-Robots-Tag on a path that robots.txt also disallows, which means the crawler never sees the header and the resource can still surface in results.
• Assuming a CMS-level robots meta tag setting also covers PDFs and images served from the same domain; it does not, those need their own server rule.
• Writing a regex pattern broad enough to catch file types that were meant to stay indexable.
• Forgetting that a CDN in front of the origin server can strip or override response headers, so the rule set in .htaccess never reaches the actual visitor.