Robots.txt and Meta Robots: Complete SEO Reference

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

This is the default for most marketing sites. User-agent: * applies to every crawler. Allow: / permits every path. The Sitemap line is not a crawl directive but a discovery hint and is read by all major crawlers.

Each group of rules begins with a User-agent line. The match is case-insensitive and uses longest-match-wins. A crawler reads only the most specific matching group and ignores other groups entirely, so splitting rules across multiple generic and specific groups rarely behaves as expected.

User-agent: Googlebot
Disallow: /private/

User-agent: Bingbot
Disallow: /private/
Disallow: /beta/

User-agent: *
Disallow: /internal/

Disallow blocks a path; Allow grants an exception within a blocked path. Both take a URL path starting with /. Empty Disallow: allows everything for that group. More specific (longer) rules win regardless of order in the file.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /search
Disallow: /cart
Disallow: /checkout

One or more Sitemap lines declare sitemap locations. These are independent of User-agent groups and can appear anywhere in the file. Always use an absolute URL. Multiple sitemaps are valid for sites with separate product, blog, and image sitemaps.

Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-products.xml
Sitemap: https://www.example.com/sitemap-blog.xml

Crawl-delay sets seconds between requests. Bing, Yandex, and Seznam honor it. Google ignores crawl-delay entirely and instead manages rate through Search Console settings, response status codes, and server response times. If Googlebot is overwhelming your origin, return 503 with a Retry-After header rather than adding crawl-delay.

Anything after # on a line is a comment. Useful for documenting why a rule exists so the next engineer does not remove it in a cleanup pass.

# Block faceted navigation parameters to save crawl budget
User-agent: *
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?sort=

Use * to match any sequence. This is most useful for query parameters and dynamic path segments. Note that / in the URL is matched by * too.

User-agent: *
# Block every URL with a session ID parameter
Disallow: /*?sessionid=
Disallow: /*&sessionid=

# Block any path containing /preview/
Disallow: /*/preview/

# Block tracking parameters globally
Disallow: /*?utm_
Disallow: /*&utm_

$ marks the end of the URL string. Without it, a pattern matches URLs that start with the prefix and have anything afterward. This catches out anyone trying to block only PDFs or only a specific file extension.

User-agent: *
# Block every PDF anywhere on the site
Disallow: /*.pdf$

# Block query-stringless print pages but not print?query=true
Disallow: /*/print$

# Combine: block any URL ending in .xml except the sitemap
Disallow: /*.xml$
Allow: /sitemap.xml$

Googlebot renders pages before indexing. If CSS or JavaScript is disallowed, Google sees an unstyled or broken version and may rank the page lower or classify it as cloaking. Explicitly allow asset paths your framework uses.

User-agent: *
Disallow: /internal/

# Always permit rendering assets
Allow: /*.css$
Allow: /*.js$
Allow: /_next/static/
Allow: /static/
Allow: /wp-content/themes/
Allow: /wp-includes/js/

When Disallow and Allow both match a URL, Google picks the rule with the longer character count. Wildcards are counted as a single character. This differs from regex engines and frequently surprises developers writing their first rules.

User-agent: *
Disallow: /products/       # 10 chars
Allow: /products/featured/ # 19 chars — WINS for /products/featured/shoe

Result: /products/shoe is blocked, /products/featured/shoe is allowed.

Without a meta robots tag, Google defaults to index, follow — the page can be indexed and links can pass PageRank. Adding noindex removes the page from the index; adding nofollow prevents link equity from flowing through any on-page links.

<!-- Default behavior, no tag needed -->
<meta name="robots" content="index, follow">

<!-- Remove from index, still crawl links -->
<meta name="robots" content="noindex, follow">

<!-- Do not pass PageRank through links -->
<meta name="robots" content="index, nofollow">

<!-- Full block from index and no link equity -->
<meta name="robots" content="noindex, nofollow">

Replace name="robots" with a specific user-agent to target only that crawler. Conflicting directives use the most restrictive combination across all applicable tags.

<!-- Applies to all crawlers -->
<meta name="robots" content="noindex">

<!-- Googlebot only -->
<meta name="googlebot" content="noindex">

<!-- Block only Google AI training (Google Search still indexes) -->
<meta name="Google-Extended" content="noindex">

noarchive prevents Google from offering a cached copy of the page. nosnippet hides the text snippet and the video preview in search results, leaving only the title and URL. Publishers use these to prevent content from appearing in AI Overviews while remaining indexable.

<!-- Index the page, no cached copy -->
<meta name="robots" content="noarchive">

<!-- Index the page, no snippet shown in results or AI features -->
<meta name="robots" content="nosnippet">

These three directives give precise control over snippet length, image preview size, and video preview duration. max-snippet takes a character count (-1 for unlimited, 0 to disable). max-image-preview accepts none, standard, or large. max-video-preview takes seconds (-1 for unlimited, 0 for none).

<!-- Recommended for editorial content -->
<meta name="robots" content="max-snippet:-1, max-image-preview:large, max-video-preview:-1">

<!-- Restrict to short snippets and no large images -->
<meta name="robots" content="max-snippet:160, max-image-preview:standard">

Tells Google to drop the URL from the index after a specific RFC 850 date. Useful for time-limited offers, event pages, or seasonal campaigns. Google stops showing the URL in Search roughly 24 hours after the date passes.

<meta name="robots" content="unavailable_after: 2026-12-31T23:59:59+00:00">

X-Robots-Tag: noindex
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: googlebot: noindex, nosnippet
X-Robots-Tag: unavailable_after: 2026-12-31T00:00:00+00:00

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

<FilesMatch "\.(doc|docx|xls|xlsx)$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

# Block PDF indexing site-wide
location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex, nofollow";
}

# Block the entire staging subdomain
server {
    server_name staging.example.com;
    add_header X-Robots-Tag "noindex, nofollow" always;
    # ... rest of config
}

// next.config.ts
export default {
  async headers() {
    return [
      {
        source: "/internal/:path*",
        headers: [
          { key: "X-Robots-Tag", value: "noindex, nofollow" },
        ],
      },
      {
        source: "/(.*\\.pdf)",
        headers: [
          { key: "X-Robots-Tag", value: "noindex" },
        ],
      },
    ];
  },
};

Googlebot fetches the raw HTML first and extracts links, canonicals, and the initial meta robots tag from the static markup. The page is then queued for rendering in a headless Chromium pool. Rendering can take seconds to days depending on load. If the meta robots tag only exists after JavaScript-execution, the first-pass indexing decision is made from the initial HTML — usually with index, follow by default.

Server-rendered frameworks like Next.js, Nuxt, Remix, and SvelteKit include meta robots in the initial HTML response. This is the only reliable way to signal noindex to Googlebot.

// Next.js app router — per-route noindex
export const metadata = {
  robots: {
    index: false,
    follow: true,
    nocache: true,
  },
};

// Produces in the HTML head:
// <meta name="robots" content="noindex,nocache" />

Libraries like React Helmet or Vue Head mutate the document head after mount. Googlebot does honor the rendered-HTML meta robots eventually, but the first crawl uses raw HTML. If your site only adds noindex via JavaScript, expect a lag of days or weeks before pages leave the index.

// React Helmet — runs only after client mount
import { Helmet } from "react-helmet-async";

function PrivatePage() {
  return (
    <>
      <Helmet>
        <meta name="robots" content="noindex" />
      </Helmet>
      <Dashboard />
    </>
  );
}

// Raw HTML initially has no meta robots tag.
// Googlebot indexes the route on first crawl, then updates later.

A classic SPA bug: the server returns 200 for every route and the client decides whether to show content or an error. If a user hits a deleted URL, the client renders a friendly "Not found" message but the HTTP status is still 200. Google flags these as soft 404s and wastes crawl budget. Fix by routing unknown paths to a URL that genuinely returns 404, or by pre-rendering known routes and returning proper status codes server-side. See our HTTP status codes reference for the full breakdown.

If robots.txt blocks a URL, Googlebot cannot fetch the page, which means it cannot read the meta robots tag. The URL remains eligible for indexing via external links and shows up in Search with no snippet. To guarantee a URL is de-indexed, allow crawling and apply noindex via meta tag or X-Robots-Tag. Once the page is removed from the index, disallow can be added back if crawl budget is a concern.

rel=canonical tells Google that two or more URLs represent the same content and that the canonical URL should be the one indexed. The non-canonical versions remain crawlable and pass signals. noindex removes the page from the index entirely. Use canonical for duplicate URLs (parameters, print versions, mobile/desktop splits) and noindex for pages that should never rank (internal search results, user dashboards, thin tag archives).

When a URL is currently indexed and must be removed, follow this sequence.

1. Ensure robots.txt does not disallow the URL. Googlebot must be able to crawl the page to read any noindex directive.
2. Add meta robots content="noindex" or X-Robots-Tag: noindex to the response.
3. Submit the URL via Search Console Removals for a six-month temporary hide while the noindex propagates.
4. Wait for Googlebot to re-crawl. Confirm removal in URL Inspection — the Indexing state should switch to "Excluded by noindex tag."
5. Once de-indexed, optionally add the URL to robots.txt disallow to save future crawl budget. Do not add disallow before the page is out of the index.

Every major AI provider now operates a dedicated crawler for training data collection. Google-Extended controls Google Gemini and Vertex AI training and is separate from Googlebot — blocking Google-Extended does not affect Search. OpenAI operates GPTBot, Common Crawl operates CCBot, Anthropic operates ClaudeBot, Perplexity operates PerplexityBot, and Apple operates Applebot-Extended for Apple Intelligence. All respect robots.txt. AI Overviews in Google Search are powered by Google-Extended-trained models plus live retrieval, so nosnippet also matters for controlling what surfaces.

# Block all major AI training crawlers
User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Keep normal search indexing open
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml