RAG System Production: 30/60/90-Day Plan 2026

Production RAG is quality engineering — and quality engineering needs a phased plan. The 30/60/90-day playbook below moves a retrieval-augmented generation system from raw corpus to production-grounded answers across three deliberate phases: ingestion and embedding choice in month one, re-ranking and faithfulness evaluation in month two, observability and refresh cadence in month three.

What's at stake: most teams ship a RAG prototype in a week and then spend six months discovering that prototype-grade retrieval, no eval suite, no citation UI, and no refresh story do not survive contact with real users. The work below is the difference between a demo and a production system — and it compounds, because each phase's artefacts (chunker, eval suite, observability) become the substrate for the next.

This guide covers seven sections: why 90 days is the right horizon, the milestones in each of the three monthly phases, how to pick between Promptfoo / DeepEval / RAGAS for the eval harness, copy-pasteable templates for the ingestion pipeline and refresh playbook, and four production failure modes worth designing against before they happen.

Ninety days is the horizon at which a competent product team can ship a RAG system that earns user trust without cutting corners anywhere meaningful. Shorter and at least one of ingestion, evaluation, or observability gets dropped — and the dropped one almost always evaluation, which is the one you cannot recover after launch. Longer and the work fragments across roadmap cycles and ships unevenly. The 30/60/90 split is not arbitrary; it maps cleanly to the three layers of the stack that decide whether the system is production-grade.

The mistake teams make is treating RAG as an LLM problem. The generation step is the most visible part of the answer surface but the smallest source of quality variance. Retrieval quality is the ceiling on answer quality — no model can cite a passage it never saw — and retrieval quality is decided by ingestion, chunking, embedding choice, and re-ranking, in roughly that order. The plan below front-loads exactly that work.

The forward projection that matters: teams that ship a phased RAG system on this horizon tend to compound. The eval harness from month two becomes the CI guardrail for month four's new embedding model. The observability work from month three becomes the substrate for incident response. The refresh cadence becomes the operational pattern that keeps quality from drifting. None of that compounding happens if the quality discipline is skipped.

For teams choosing between this phased plan and a faster ship-it-and-iterate approach: the faster approach works for internal prototypes and demos but does not produce a system that survives the third week of real user traffic. The phased approach takes ninety days; the unstructured approach usually takes six months and produces a system the team cannot confidently iterate on. Pick the discipline up front.

Month one builds the foundation. The deliverable is a working corpus → embeddings → retrieval pipeline that recalls the right passages for a representative query set — even without a polished answer UI on top. The five milestones below are the critical path; skipping any of them produces problems that cost weeks to recover from later.

The single most under-invested step in month one is the recall baseline. Teams treat it as optional because it requires human labelling — fifty queries, an afternoon of work — and the labelled set then sits as the foundation every subsequent change is judged against. Without it the team has no way to tell whether week-five's clever chunker tweak actually helped or just felt like it should have.

For the underlying stack — Postgres + pgvector, the canonical three-table schema, ingestion and retrieval SQL — our self-hosted RAG tutorial walks through the implementation in detail. This playbook stays at the planning layer; the tutorial is the paired step-by-step.

Month two converts a retrieval pipeline into a system that produces grounded answers users can trust. Three concerns dominate: lifting recall on hard queries with hybrid + re- ranking, instrumenting faithfulness so hallucination is measurable, and shipping the citation UI that turns "the model said" into "the model said because this passage says".

The most consequential of the five milestones is week seven's faithfulness eval. Without it the team has no way to know whether a prompt-template tweak or a new chunker improved or degraded answer quality on real distribution. With it, every subsequent change is measurable — and the CI pipeline can fail a deploy that regresses faithfulness below a threshold. That guardrail is what makes the system iterable for months and years rather than just weeks.

Month three is the operational phase. The system promotes to production behind staged rollout, every query path is instrumented, and the refresh cadence that prevents quality drift is scheduled and rehearsed at least once before the quarter ends. The five milestones below close out the playbook.

The hidden milestone in month three is week eleven's refresh cadence. Most teams ship without one and then watch answer quality degrade silently over the next four to six months as the corpus drifts away from what the embeddings were trained on. The fix is operational, not technical: schedule the incremental refresh, gate it on checksum changes so the embedding bill stays bounded, and rehearse the full re-embed at least once before you actually need it.

Three eval frameworks dominate the production RAG toolbox in 2026 — Promptfoo, DeepEval, and RAGAS. They are not interchangeable; each is shaped for a different team archetype. Pick the one whose archetype matches yours; do not run all three. The maintenance overhead of multiple eval harnesses defeats the discipline they are meant to enforce.

The decision rule that holds up: if your team is mostly TS/JS and your retrieval lives in the same repo as your application, Promptfoo is the lowest-friction pick. If your team is mostly Python and you already write pytest-based suites for the rest of your ML stack, DeepEval slots into the existing test discipline. If RAG is the product and you need the standard metric vocabulary for stakeholder reporting, RAGAS is the right pick. The wrong move is picking based on stars or recency — pick based on archetype.

Three reusable templates anchor the playbook. The ingestion CLI is the entry point for adding or updating sources; the eval CI step is what guards faithfulness regressions; the refresh cron is what keeps the corpus in sync without a human in the loop. Copy them, rename them, ship them.

Ingestion CLI

// scripts/ingest.ts — drop a source URL or local path, get checksum-gated re-embed
import { readSource } from '@/lib/rag/source';
import { chunkBySlidingWindow } from '@/lib/rag/chunk';
import { embedBatch } from '@/lib/rag/embed';
import { upsertDocument } from '@/lib/rag/upsert';

async function main() {
  const sourceUrl = process.argv[2];
  if (!sourceUrl) throw new Error('Usage: pnpm ingest <source-url>');

  const { title, mimeType, text } = await readSource(sourceUrl);
  const chunks = chunkBySlidingWindow(text, { size: 600, overlap: 50 });
  const embeddings = await embedBatch(chunks.map((c) => c.content));

  await upsertDocument({
    sourceUrl,
    title,
    mimeType,
    content: chunks.map((c) => ({
      ord: c.ord,
      text: c.content,
      tokens: c.tokenCount,
    })),
    embeddings,
  });

  console.log(`Ingested ${sourceUrl} · ${chunks.length} chunks`);
}

main().catch((e) => {
  console.error(e);
  process.exit(1);
});

Eval CI step

# .github/workflows/rag-eval.yml — nightly faithfulness regression gate
name: RAG eval

on:
  schedule:
    - cron: '0 3 * * *'  # nightly at 03:00 UTC
  workflow_dispatch:

jobs:
  faithfulness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v3
      - run: pnpm install --frozen-lockfile

      # Run the labelled eval set through the RAG pipeline and score
      - run: pnpm eval:rag --set eval/v2.jsonl --out reports/eval.json

      # Fail the job if faithfulness drops below the threshold
      - run: |
          node -e "
            const r = require('./reports/eval.json');
            if (r.faithfulness < 0.85) {
              console.error('faithfulness below threshold:', r.faithfulness);
              process.exit(1);
            }
          "

      - uses: actions/upload-artifact@v4
        with:
          name: rag-eval-report
          path: reports/eval.json

Refresh playbook

// scripts/refresh.ts — incremental, checksum-gated, idempotent
import { listSources } from '@/lib/rag/registry';
import { fetchSource } from '@/lib/rag/source';
import { ingestIfChanged } from '@/lib/rag/upsert';

const MAX_PARALLEL = 8;

async function main() {
  const sources = await listSources();
  let touched = 0;
  let skipped = 0;

  for (let i = 0; i < sources.length; i += MAX_PARALLEL) {
    const batch = sources.slice(i, i + MAX_PARALLEL);
    const results = await Promise.allSettled(
      batch.map(async (s) => {
        const fresh = await fetchSource(s.url);
        const changed = await ingestIfChanged(s, fresh);
        return changed ? 'touched' : 'skipped';
      }),
    );
    for (const r of results) {
      if (r.status === 'fulfilled' && r.value === 'touched') touched++;
      else if (r.status === 'fulfilled') skipped++;
    }
  }

  console.log(`Refresh complete · touched=${touched} skipped=${skipped}`);
}

main().catch((e) => {
  console.error(e);
  process.exit(1);
});

Two operational notes. First, the eval CI job should run on a schedule, not on every PR — the labelled set takes minutes to score and the relevant signal is daily drift, not per-commit regression. Second, the refresh script should write its touched/skipped counts to your observability surface; an unexpected zero on the touched count usually means a source registry pointing at a dead URL.

Four failure modes recur across production RAG deployments. None of them are exotic; all of them are avoidable with a single deliberate design choice up front. The bars below quantify how often each appears in systems built without the phased plan above — order-of-magnitude indicative, not a controlled study.

1. Silent corpus drift

Source documents change, the re-ingest job either does not exist or runs without a checksum gate, and answer quality degrades silently over months. Mitigation is exactly the week-eleven refresh cadence — incremental, checksum-gated, scheduled, observable.

2. Hallucination on out-of-corpus questions

User asks a question the corpus doesn't cover, the model answers anyway from training data, the answer is confidently wrong with a citation that's only tangentially related. Mitigation is the week-eight hallucination guard — distance threshold on the top retrieved chunk, explicit refusal copy when the threshold is breached.

3. Recall regression on a model upgrade

New embedding model ships, team upgrades without re-running the eval set, recall@10 drops three points, faithfulness tanks two weeks later when users start noticing. Mitigation is the week-seven eval suite plus the CI gate from the template above — no model change ships without a passing eval.

4. Citation UI as launch polish

Citations get pushed to "v2" or "after launch", users have no way to verify the model's claims, trust collapses inside the first month. Mitigation is the week-eight citation UI — built into the answer surface from day one of public access, not bolted on after.

If you are looking for an outside-in version of this same discipline as a one-page audit, the companion 80-point RAG quality scorecard scores an existing deployment against the same dimensions this plan builds toward. The two pair cleanly — the plan builds the system; the scorecard audits it once it's live.