Build an Agentic SEO Crawler with Claude Code

An agentic SEO crawler is the bridge between data extraction and editorial judgment — a Playwright-backed Claude Code workflow that queues URLs, audits each one for twelve on-page signals, and synthesises the findings into a severity-weighted, fix-oriented report. This tutorial walks the full build, end to end.

Static crawlers — Screaming Frog, Sitebulb, the SaaS clones — still dominate SEO toolchains because they are fast, deterministic, and exhaustive. They are also brittle in exactly the places that matter most in 2026: JavaScript-rendered SPAs, schema correctness, and the gap between "present" and "useful." A title tag can exist and still be wrong. A meta description can be the right length and still cannibalise three other pages. A structured-data block can validate and still misrepresent the entity. Static crawlers find the absence; agentic crawlers judge the substance.

What follows is a working build. The full pattern is three subagents coordinated by a single Claude Code session: an orchestrator that queues URLs and synthesises results, an auditor that drives Playwright and extracts twelve structured signals, and a reporter that ranks findings by severity and emits a fix-oriented markdown report. You will see the agent definition files, the orchestration script, a real audit run, and the CI wiring that turns one-time runs into a continuous signal.

Screaming Frog crawls 500 URLs per minute and dumps a 40-column CSV. Sitebulb adds visual hints and crawl-graph diagrams. Both are excellent at what they do — surface the absence of a thing, count the things that are present, and compare against a known schema of rules. They are the reason most agencies still checklist SEO instead of judging it.

What they cannot do is judge whether a title is good. They cannot tell you that a meta description, while technically 152 characters and unique, sells the wrong intent. They cannot read a schema block and notice that the Product entity points to a review aggregate from a different SKU. They cannot read a paragraph and observe that it answers a question the H1 does not pose. Those are the findings that move rankings, and they are precisely where a static crawler is silent.

Agentic crawlers add three capabilities a static crawler cannot: semantic judgment (is this title good, not just present), contextual scoring(does this signal matter for this page's job), and narrative synthesis (what does the pattern across 50 pages tell us about the site). The job is not to replace the static crawler. The job is to read what the static crawler missed.

The most useful framing is not "agentic vs static." It is "static feeds the agent." Static crawlers remain the right tool for the URL inventory, the indexability check, and the structural audit. The agentic layer sits on top, taking the shortlist of pages where editorial judgment is the binding constraint, and producing the report a senior SEO would have written by hand.

The architecture is deliberately small. One Claude Code session acts as the orchestrator — it owns the URL frontier, the concurrency limit, the results aggregation, and the final hand-off. Two Claude Code subagents do the specialised work: an auditor that drives Playwright, and a reporter that turns the auditor's structured output into a ranked narrative. Subagents are defined as plain markdown files in .claude/agents/, each with a YAML frontmatter block (name, description, allowed tools, model) and a system prompt below.

The decision to put Playwright inside the auditor subagent rather than the orchestrator is deliberate. The orchestrator runs once per audit. The auditor runs once per URL — sometimes 50 times, sometimes 5,000. Putting the browser inside the per-URL agent means concurrency is a configuration value (run six auditors in parallel, or twenty), not a re-architecture. It also means each audit gets its own isolated browser context, so cookies and local storage do not leak across URLs and falsify the rendered DOM.

The orchestrator is a single Node script — no framework, no queue, no Redis. It expands the seed URL into a frontier, enforces a configurable concurrency limit, calls the auditor subagent once per URL, aggregates the resulting JSON, and finally invokes the reporter. A reasonable starting shape, with the boring parts elided:

// scripts/audit.mjs — orchestrator entrypoint
import { execFile } from "node:child_process";
import { readFile, writeFile, mkdir } from "node:fs/promises";
import { promisify } from "node:util";

const run = promisify(execFile);
const SEED = process.argv[2];                  // npm run audit -- https://example.com
const CONCURRENCY = Number(process.env.CRAWL_CONCURRENCY ?? 6);
const MAX_URLS = Number(process.env.CRAWL_MAX ?? 50);

// 1. Expand the frontier from sitemap.xml (fallback: shallow same-host crawl)
const frontier = await expandFrontier(SEED, MAX_URLS);

// 2. Concurrency-limited dispatch — N workers pull from the queue
const findings = [];
const workers = Array.from({ length: CONCURRENCY }, () => worker());
await Promise.all(workers);

async function worker() {
  while (frontier.length) {
    const url = frontier.shift();
    if (!url) return;
    const { stdout } = await run(
      "claude",
      ["--agent", "seo-auditor", "--print", url],
      { maxBuffer: 50 * 1024 * 1024 }
    );
    findings.push(JSON.parse(stdout));
  }
}

// 3. Hand the merged JSON to the reporter for ranking + synthesis
await mkdir(".audit", { recursive: true });
await writeFile(".audit/findings.json", JSON.stringify(findings, null, 2));

const { stdout: report } = await run(
  "claude",
  ["--agent", "seo-reporter", "--print", ".audit/findings.json"],
  { maxBuffer: 50 * 1024 * 1024 }
);

await writeFile(".audit/report.md", report);
console.log("\n✓ Report written to .audit/report.md");

A few choices worth flagging. The frontier is a plain array — push to it from sitemap.xml, shift from it in the worker. For sites under ~5,000 URLs that is plenty; for larger crawls swap to a persistent queue (SQLite, BullMQ) so a crashed run resumes cleanly. Concurrency is a simple worker pool — six is the right default for most laptops, twenty if you are running on a server with adequate egress. The subagent invocation uses claude --agent <name> --print <input> which streams a structured response to stdout that the orchestrator parses as JSON.

The biggest production gotcha is rate-limiting the target. A 20-way concurrent crawl against a small WordPress site is a denial of service. The orchestrator should respect robots.txt, honour Crawl-Delay headers, and default to a conservative six workers. Production engagements typically wire in a token-bucket limiter keyed by hostname — outside the scope of this tutorial, but a one-screen addition to the worker function.

The auditor subagent lives at .claude/agents/seo-auditor.md. Its single responsibility is to render one URL with Playwright and emit a JSON document describing the twelve on-page signals. It never judges, never narrates, never ranks — those are the reporter's jobs. Keeping the auditor strictly structural is what allows you to swap the model behind it (Sonnet → Haiku for cost, Opus for hard-to-render pages) without touching the orchestrator.

---
name: seo-auditor
description: Audits a single URL — renders with Playwright, extracts 12 on-page SEO signals, emits strict JSON. Invoked per URL by the orchestrator.
tools: Read, Write, Bash
model: sonnet
---

You are the seo-auditor subagent. You audit exactly one URL per
invocation and return a strict JSON document. You never write prose,
never rank, never recommend — that is the reporter's job.

## Workflow on every invocation

1. Read the URL passed as your input (a single string).
2. Drive Playwright via the included extract.mjs helper:
   `node scripts/extract.mjs <url> > /tmp/raw.json`
   The helper renders the page headless, waits for networkidle,
   and dumps the hydrated DOM + headers + timing + structured data.
3. Read /tmp/raw.json and compute the 12 signals below.
4. Emit a single JSON document to stdout matching the schema:

   {
     "url": "<input>",
     "fetched_at": "<ISO 8601>",
     "signals": {
       "title":           { "value": "...", "length": N },
       "meta_description":{ "value": "...", "length": N },
       "h1":              { "values": ["..."], "count": N },
       "schema":          { "types": ["..."], "valid": boolean, "errors": [] },
       "canonical":       { "value": "...", "self_referential": boolean },
       "hreflang":        { "entries": [{ "lang": "...", "href": "..." }] },
       "image_alts":      { "total": N, "missing": N, "empty": N },
       "internal_links":  { "total": N, "unique": N, "anchor_quality": "good|mixed|poor" },
       "word_count":      { "value": N },
       "render_time_ms":  { "value": N },
       "cwv_proxies":     { "lcp_ms": N, "cls": N, "fid_proxy_ms": N },
       "structured_data": { "blocks": N, "json_ld_present": boolean }
     },
     "raw": { "status": N, "redirects": N, "headers": { ... } }
   }

## Hard rules

- Output JSON only. No prose, no comments, no markdown fences.
- Every signal MUST be present even if empty (use null or [] / {}).
- Never editorialise. `anchor_quality` is the one judged field —
  use it sparingly: "good" if anchors describe targets, "mixed" if
  half are generic, "poor" if >75% are "click here" / "read more".
- If Playwright fails, emit { "url": "...", "error": "<message>" }
  and exit 0. The orchestrator handles partial runs.

The model choice is worth a brief note. Sonnet is the production default — it parses the rendered DOM, runs the signal computation, and emits the JSON in a single pass at modest cost. Haiku works for sites with simple markup and tight budgets but produces slightly less reliable schema validation. Opus is overkill for the extraction step and should be reserved for sites with intentionally obfuscated markup or unusual schema patterns. The trinity is designed precisely so you can route by complexity.

The reporter is the synthesis layer. It receives the merged JSON for every URL in the audit, applies a severity weighting to each finding, groups by impact, and emits a markdown report with code-fix snippets. The reporter never opens a browser; the reporter never re-fetches a URL. It works exclusively from the auditor's structured output, which keeps the synthesis step deterministic and inexpensive.

---
name: seo-reporter
description: Synthesises auditor JSON into a severity-ranked markdown report. Invoked once per audit run by the orchestrator.
tools: Read, Write
model: sonnet
---

You are the seo-reporter subagent. You synthesise the auditor's
structured findings into a markdown report ranked by severity. You
never open a browser, never re-fetch, never re-audit. You work
exclusively from the JSON the orchestrator hands you.

## Workflow on every invocation

1. Read the findings file path passed as your input.
2. Apply the severity matrix (see below) to each signal across each URL.
3. Group findings by severity bucket: critical, high, medium, low.
4. For each finding, generate a one-line summary and a code-fix snippet.
5. Emit a markdown report with this exact structure:

   # SEO Audit — <site> — <date>
   <one-paragraph executive summary>

   ## Top actions (this sprint)
   1. <critical finding 1>
   2. <critical finding 2>
   3. <high finding>
   ...

   ## Critical findings
   ### <Finding title>
   **Affects:** <N URLs>
   **Why it matters:** <2 sentences>
   **Fix:**
   ```html
   <code snippet>
   ```
   **URLs:** <bullet list>

   ## High findings
   ...

   ## Medium findings
   ...

   ## Low findings
   ...

## Severity matrix (apply uniformly)

- critical — blocks indexing or causes wrong content to surface
  (missing/non-self-referential canonical, invalid schema on commerce
  pages, missing/duplicate H1, robots noindex on revenue pages).
- high — measurable ranking impact within one quarter
  (titles >60 chars, descriptions outside 120-160, hreflang errors,
  image alt missing on >20% of images, LCP >2.5s).
- medium — quality issues that compound over time
  (poor anchor quality on internal links, thin content <300 words,
  schema present but minimal, no structured data on eligible page).
- low — polish, monitoring
  (description slightly long, single missing alt, marginal CLS).

## Hard rules

- Top actions list is ≤7 items and only includes critical + high.
- Every finding section MUST include a code-fix snippet.
- Cite URLs by bullet list under each finding, capped at 10 per finding.
- Tone: terse, technical, no marketing language.

The reporter's most important responsibility is the Top actions list at the head of the report. Without it, an SEO lead reading the report has to read the whole thing to know what to do first. With it, the first paragraph of the report is already a sprint plan. That single design choice is what turns the agentic crawler from a data pipeline into a decision-support tool.

The twelve signals are deliberately narrow. The audit is not trying to catalogue every possible on-page attribute; it is trying to surface the signals most correlated with ranking outcomes and most often broken in real production sites. The severity weighting below is what the reporter applies — keep it consistent with the matrix in the reporter system prompt.

Notice that several of the twelve are not signals you would find in a static crawler at all — anchor quality, schema validation against entity correctness, render-time-from-rendered-DOM. Those are the additions that justify the agentic layer. Everything else is the baseline a static crawler would cover too; the reason it is in this audit is so the orchestrator can run a single tool against an unknown site and produce a complete picture, not so the agentic crawler replaces the static one.

With the orchestrator, auditor, and reporter wired up, the audit is a single npm script. The seed URL is the only required input; concurrency and the crawl ceiling are environment variables with sensible defaults.

// package.json
{
  "scripts": {
    "audit": "node scripts/audit.mjs",
    "audit:ci": "CRAWL_MAX=200 CRAWL_CONCURRENCY=10 node scripts/audit.mjs"
  }
}

A first run looks like this in the terminal:

$ npm run audit -- https://example.com

> agentic-seo-crawler@0.1.0 audit
> node scripts/audit.mjs https://example.com

[orchestrator] expanding frontier from sitemap.xml...
[orchestrator] frontier: 50 URLs (ceiling: 50)
[orchestrator] dispatching with concurrency 6
[auditor]  ✓ https://example.com/ — 12 signals, 1.8s
[auditor]  ✓ https://example.com/pricing — 12 signals, 2.1s
[auditor]  ✓ https://example.com/docs — 12 signals, 1.6s
[auditor]  ⚠ https://example.com/blog/old-post — schema invalid
[auditor]  ✓ https://example.com/contact — 12 signals, 1.4s
... (45 more)
[orchestrator] all auditors complete in 3m 47s
[reporter] synthesising 600 signal-points across 50 URLs...
[reporter] severity breakdown: 4 critical, 12 high, 23 medium, 31 low
[reporter] report written to .audit/report.md

✓ Done in 4m 12s. Cost: $0.41

And a representative excerpt from the resulting report:

# SEO Audit — example.com — May 2, 2026

Audited 50 URLs across the example.com host. The report surfaces 4
critical, 12 high, 23 medium, and 31 low findings. The top three
actions below account for ~80% of the projected ranking impact.

## Top actions (this sprint)

1. Add self-referential canonical to /pricing, /docs, /contact (3 URLs).
2. Fix invalid schema on /blog/old-post — Article type missing
   datePublished and author.
3. Reduce LCP on /pricing from 4.1s → under 2.5s (largest contentful
   element is the hero image, no width/height, no priority loader).

## Critical findings

### Missing canonical on indexable pages
**Affects:** 3 URLs
**Why it matters:** Without a canonical, Google chooses the canonical
itself — usually correctly, but exposing you to duplicate-content
clustering when query parameters or trailing slashes drift.
**Fix:**
```html
<link rel="canonical" href="https://example.com/pricing" />
```
**URLs:**
  - https://example.com/pricing
  - https://example.com/docs
  - https://example.com/contact

### Invalid Article schema on /blog/old-post
**Affects:** 1 URL
...

The pattern is composable by design. Adding a fourth subagent — for accessibility, Core Web Vitals from CrUX, internal-link graph analysis, or domain-specific commerce signals — does not require touching the orchestrator. Drop a new .md file in .claude/agents/, give it the same JSON contract (input: URL, output: findings JSON), and add one line to the orchestrator's dispatch loop. The reporter consumes the additional findings the same way it consumes the original twelve signals — by severity, with the matrix you extended for the new dimensions.

The CI wiring is similarly small. A GitHub Action runs the audit on a weekly cron, diffs the new report against the prior week, and posts the delta — new critical findings, resolved findings, net change in severity counts — to a Slack channel. Two failure modes are worth handling explicitly: a new critical finding should block a release branch from merging until the team acknowledges it, and a previously-resolved finding that regresses should ping the author of the commit that introduced it.

# .github/workflows/seo-audit.yml
name: Agentic SEO Audit
on:
  schedule:
    - cron: '0 6 * * 1'        # Mondays 06:00 UTC
  workflow_dispatch:

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npx playwright install chromium
      - name: Run audit
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          CRAWL_MAX: '200'
          CRAWL_CONCURRENCY: '10'
        run: npm run audit:ci -- https://example.com
      - name: Diff against last week
        run: node scripts/diff-reports.mjs
      - name: Post to Slack
        env:
          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
        run: node scripts/notify-slack.mjs

The compounding value of the CI integration is not the weekly report — it is the regression catch. A change in a CMS template that strips a canonical, an A/B test that ships a new H1 with the wrong intent, a launch that ships pages with missing schema: without the weekly audit those changes surface in organic-traffic decline three months later. With the weekly audit, they surface in Slack the following Monday. The same agentic pattern works for any audit workflow that combines extraction, judgment, and synthesis — accessibility, performance, content quality, brand compliance. The trinity is the reusable template; the signals are the customisation. For deeper coverage of the broader agentic-SEO methodology, see our crawl-to-implementation playbook and the step-by-step custom subagent guide. Teams building the wider audit infrastructure also work against the 200-item technical SEO checklist and our agent-first marketing stack audit.