The 302 Wall: Why AI Crawlers can’t see Your Archive

An AI crawler discoverability audit answers one deceptively simple question: when a machine — not a human — requests your pages, what does it actually get back? On a recent engagement with an independent writer’s publication, the answer was alarming. Every single archive post, a multi-hundred-post body of work, returned an HTTP 302 redirect to the publication’s welcome page when fetched by a crawler.

The stakes are specific to 2026. Common Crawl’s public web archive is among the most widely documented raw sources feeding LLM training corpora, and AI assistants increasingly ground their answers in live crawler fetches. Content that answers crawlers with a redirect instead of HTML is not under-optimized — it is absent. No amount of on-page work, schema markup, or llms.txt architecture matters if the crawler never receives the article in the first place.

This playbook walks through the finding, the redirect semantics that make a 302 wall so damaging, the detection method (one curl command), the custom-domain fix that moved every post from 302 to 200, the companion platform settings that complete AI discoverability, and a generic five-minute test you can run on any hosted platform today — Substack, Ghost, Medium, or a headless CMS sitting behind an auth wall.

The engagement was a pre-launch audit for an independent author’s publication hosted on Substack. In a browser, everything looked healthy: posts loaded, the archive paginated, nothing suggested a problem. The discoverability audit started the way ours always do — by requesting pages the way machines do rather than the way people do.

The first crawler-agent fetch came back wrong. Instead of the article, the response was a 302 redirect pointing at the publication’s /welcome page. The second post did the same. So did every other post we sampled across the archive. The pattern was uniform: browser User-Agents received the article; crawler User-Agents received a redirect to the same welcome page, regardless of which post was requested.

The implication compounds with archive size. One redirected page is a glitch; an entire archive of several hundred posts redirecting to a single signup page means the publication’s whole body of work was contributing nothing to the web crawls that AI systems learn from and ground against. Years of writing, effectively absent from the machine-readable web.

HTTP redirect codes are instructions to the requester, and crawlers take them literally. The two that matter here carry opposite meanings. A 301 (Moved Permanently) says: the content you want now lives at the target URL — follow it, index the target, and consolidate signals there. A 302 (Found, temporary) says: the content is momentarily somewhere else, keep treating the original URL as canonical, and expect the situation to change.

A blanket 302 wall breaks both halves of that contract. The redirect target is not the content — it is a generic welcome page, identical for every URL requested. So a crawler following the wall dutifully arrives at the same signup page hundreds of times, and the archive it came for never appears in any response body. Search engines have heuristics for long-standing 302s (Google has said it eventually treats persistent 302s much like 301s), but no heuristic can recover article text that was never served.

The cruelest property of this failure mode is its invisibility to the owner. Everything a human checks — the homepage, the archive page, individual posts, even mobile rendering — works perfectly, because the wall keys on the User-Agent header, not on anything a person can see. The only way to catch it is to ask for your own pages the way a machine does. That is also why we treat crawler-agent fetches as step one of any discoverability audit, the same way server log analysis is step one of any crawl-budget audit: both look at what machines actually experienced, not what the CMS claims.

Why does one crawler’s view of your archive matter so much? Because of where that view flows. Common Crawl is a nonprofit that has been running large periodic crawls of the public web for over a decade and publishing the results as free, open datasets. Its crawler identifies itself with a documented User-Agent — CCBot — which is exactly why we impersonate it during audits: it shows us what Common Crawl would have received.

Filtered derivatives of Common Crawl’s archive are among the most widely documented ingredients in LLM training mixes, and the dataset’s role in AI training pipelines is central enough that it has become a legal battleground — publishers are actively fighting over what sits in that archive. The practical takeaway for a content owner is simpler than the litigation: if CCBot receives a redirect instead of your article, your writing is absent from one of the primary reservoirs that AI systems draw on.

Training corpora are only half the exposure. AI assistants also fetch pages live — retrieval crawlers ground answers against current web content at question time. A User-Agent-keyed 302 wall plausibly blocks those fetches too, which means the archive was likely missing from both timescales: the slow one (corpus inclusion) and the fast one (answer-time grounding). Being present and fetchable is the entry ticket; what earns citations once you are in is a separate, later problem.

The detection method is deliberately primitive, because primitive is reproducible. You request the same URL twice — once as a browser, once as a crawler — and compare the status lines. The -I flag asks for headers only, -s silences the progress noise, and -A sets the User-Agent string.

# 1. Fetch an archive post the way a browser would
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)" \
  https://example-publication.substack.com/p/your-post-slug

HTTP/2 200
content-type: text/html; charset=utf-8

# 2. Fetch the SAME post the way Common Crawl's bot does
curl -sI -A "CCBot/2.0 (https://commoncrawl.org/faq/)" \
  https://example-publication.substack.com/p/your-post-slug

HTTP/2 302
location: https://example-publication.substack.com/welcome

That second response is the wall. The same request that earns a human a full article earns a crawler a redirect — and the location header shows where every machine visitor was being sent. On the publication we audited, this pattern held for every archive post we tested. Checking the response body confirms it: fetching with -s (without -I) and searching the output for a phrase from the article returns nothing under the crawler identity, because the article HTML is simply not in the response.

Two cautions before you treat this test as gospel. First, a redirect for crawlers is not automatically malice or even misconfiguration you control — hosted platforms route traffic through logic you cannot see, and behavior can differ by User-Agent, geography, or session state. Second, some infrastructure providers verify crawler identity by IP rather than User-Agent string, so an impersonated fetch may occasionally be treated differently from the real bot. The test is a strong first-pass signal, not a full crawl simulation — when it fails, you have found something real to investigate.

The fix that worked was mapping a custom domain to the publication. Mechanically, this is standard hosted-platform DNS: the www subdomain gets a CNAME record pointing at the platform’s custom-domain target, and the apex domain 301-redirects to www. Once the platform served the publication from its own domain, crawler fetches changed state completely.

We verified the change the same way we detected the problem — by fetching as CCBot and reading the responses. Posts on the custom domain returned HTTP 200 with the full article HTML under a crawler User-Agent. The platform-subdomain copies of the same posts now returned 301 redirects to the custom domain — the correct permanent signal, consolidating everything to a single canonical home rather than splitting the archive across two hosts.

One honesty note, because it matters for anyone applying this elsewhere: we did not reverse-engineer the platform’s internal routing to prove why subdomain-hosted publications were walling crawlers while custom-domain ones were not. Hosted platforms change routing logic without notice. What we can state is the measured before-and-after: the same crawler fetch that received a 302 wall received a 200 and full HTML after the domain mapping. That measurable state change — not a promised citation lift — is the deliverable of this kind of audit.

The custom domain also carries a strategic bonus unrelated to crawlers: it moves the publication’s accumulating search equity onto an asset the writer owns. Links, mentions, and eventual AI citations accrue to a domain that can be pointed anywhere later — at a different platform, a self-hosted stack, or a future site — instead of being locked to a subdomain the platform controls. For any serious publication, that ownership argument justifies the custom domain even before the crawler-visibility argument arrives.

The domain fix opened the door; four companion settings determine what walks through it. On Substack these are literal toggles in the publication settings, but every hosted platform has equivalents — treat this as a category checklist, not a vendor manual. One framing note: these recommendations assume the goal is AI-answer citability. A writer who wants their work excluded from AI training should set the first toggle the opposite way — that is a legitimate editorial position, just a different one, and we cover it in the AI crawler access-control decision matrix.

Everything below is generic — swap in your own URLs and run it against any hosted platform. You need a terminal with curl, three representative post URLs (one recent, one old, one paywalled if you have them), and five minutes.

Step 1 — baseline as a browser. Confirm the page works for humans and note the status line.

curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)" \
  https://www.example.com/p/sample-post
# Expect: HTTP/2 200 + content-type: text/html

Step 2 — repeat as Common Crawl. Same URL, crawler identity. This is the fetch that decides corpus eligibility.

curl -sI -A "CCBot/2.0 (https://commoncrawl.org/faq/)" \
  https://www.example.com/p/sample-post
# Healthy: HTTP/2 200
# Wall:    HTTP/2 302 + location: .../welcome (or /subscribe, /signup)

Step 3 — repeat with other AI crawler identities. Behavior can differ per bot, so sample the major documented User-Agents.

for UA in "GPTBot" "ClaudeBot" "PerplexityBot"; do
  echo "== $UA =="
  curl -sI -A "$UA" https://www.example.com/p/sample-post | head -2
done

Step 4 — verify the body, not just the headers. A 200 with an empty shell is still a fail; confirm a phrase from the article is actually in the crawler-served HTML.

curl -s -A "CCBot/2.0 (https://commoncrawl.org/faq/)" \
  https://www.example.com/p/sample-post | grep -c "a phrase from the article"
# Expect: 1 or more. Zero means the article text is not being served.

Step 5 — check the enumeration layer. Fetch /robots.txt and confirm it does not disallow the bots you want, then fetch /sitemap.xml and confirm your archive URLs are actually listed. Repeat steps 2–4 for your two other sample posts. If any step shows a browser-versus-crawler split, you have found your own 302 wall — and on a hosted platform, the custom-domain route plus the settings checklist in the previous section is the first fix to try.

Most of the AI-crawler conversation in 2026 runs in one direction: publishers deciding how much to block, with robots.txt rules, bot-management layers, and licensing standoffs. This audit is the mirror image, and the mirror is instructive. The same mechanics that let a publisher deliberately wall off AI crawlers — User-Agent detection, redirects, access gates — can wall off a writer who wants in and has no idea the wall exists. Exclusion is now the accidental default failure mode, not just a policy choice.

Our reading of the trend: as AI assistants become a primary discovery surface, discoverability audits split into two distinct disciplines. Access control — who may read you — is a policy question covered by robots directives and the emerging licensing layer. Access verification — who actually can read you — is an engineering question that almost nobody is asking, because every dashboard, preview, and human check shows a working site. We expect the second discipline to become a standard line item in technical SEO engagements, the way log-file analysis already is — and platform-hosted publications, where the owner controls neither routing nor headers, are exactly where silent failures like this one will keep surfacing.

Looking forward, the asymmetry favors whoever verifies. Writers and brands that confirm crawler readability today are positioned for whatever share of discovery AI answers end up carrying; those that never check are betting their archive on a routing decision made by a platform they do not control. This is the verification layer of our agentic SEO engagements: before any content or citation-optimization work, prove that machines receive the pages at all — then build the AEO program on top of a door that is actually open.