RAG Failure Modes: What the Published Research Documents

Retrieval-augmented generation, the pattern where a system fetches documents and hands them to a language model before it answers, fails at several stages, and the research has named them. The spine of the evidence is a 2024 experience report by Barnett and colleagues that identified seven failure points across three deployed systems. Around it sit a 2023 measurement of how position inside the context changes accuracy, a 2026 taxonomy of sixteen error types by pipeline stage, a 2026 literature review counting thirty-three modes, and one vendor's figures on how much a retrieval change helped.

The useful thing about reading these together is that they disagree on granularity and agree on structure. Whether you count seven failure points or thirty-three modes, the failures cluster into the same places: the answer is not in the corpus, the answer is in the corpus but did not rank, the answer ranked but did not survive into the context, the answer was in the context and the model did not use it, and the model used it and still produced the wrong shape. Every mitigation in the literature targets one of those places.

This post is the evidence, not a scorecard. For the metrics to run on your own system our RAG metrics guide is the companion; for the checklist, the 80-point RAG audit scorecard. Here the question is narrower: what has actually been documented, by whom, on what.

The paper that organises the field is "Seven Failure Points When Engineering a RAG System" by Barnett and colleagues, posted to arXiv on January 11, 2024 and presented at CAIN 2024. It is explicitly an experience report drawn from three case studies: a research-paper reviewer, an education tutor over 38 documents, and a biomedical question-answering system over 4,017 documents. Its two framing claims, from the abstract, set the tone for everything since: that validation of a RAG system "is only feasible during operation", and that the robustness of such a system "evolves rather than designed in at the start".

The seven points are defined in the paper's fifth section, and each is a different place in the pipeline. They are listed below in the paper's order with the paper's own definition, because later taxonomies are mostly refinements of these.

Why the list still matters is its shape, not its count. Three of the seven happen before the model sees anything and four happen inside generation. A team that only tunes the prompt is working on four of seven, and cannot touch the other three. The paper's own lessons are correspondingly spread out: calibrate chunk size and strategy, add metadata such as the file name and chunk number into the chunk because it improved retrieval, run offline evaluation and runtime monitoring together, and accept that the system "require[s] continuous calibration". Its stated limit is equally plain: three case studies, and an automated check that was more pessimistic than a human rater on the biomedical set.

FP2 is the failure most teams can measure, and its best-known numbers are a vendor's. On September 19, 2024 Anthropic published "Introducing Contextual Retrieval", describing a retrieval failure that is really a chunking failure: once a document is split, each chunk loses the context that would let an embedding or a keyword index match it. Its example is an error-code string that an embedding model can miss entirely. The remedy it describes is to prepend a short, chunk-specific context before embedding and to index that context for BM25 keyword search as well, then optionally rerank.

The figures below are Anthropic's, from internal tests across codebases, fiction, arXiv papers and science papers, using its best-performing embedding configuration and a top-20 chunk retrieval. The metric is one minus recall at twenty: the share of queries where the relevant chunk was not among the twenty returned. It measures retrieval, not whether the final answer was right, and it is vendor-run on private data with 2024 models.

Two things to carry from the chart. First, the baseline matters as much as the improvement: a 5.7% top-20 miss rate on a vendor's curated sets is already low, and a production corpus with duplicate, stale or badly split documents will start higher. Second, the same post gives advice that bypasses retrieval altogether: if the knowledge base is under about 200,000 tokens, roughly 500 pages, Anthropic suggests putting the whole thing in the prompt and not building RAG at all. Whether that is cheaper depends on caching, which our long-context retrieval test and prompt caching economics guide cover from each side.

FP3 and FP4 share a cause the research measured before Barnett named them. In "Lost in the Middle", posted July 6, 2023, Liu and colleagues ran multi-document question answering and key-value retrieval with 20 and 30 documents in the context, moving the one relevant document through every position. The abstract's finding: performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models".

The number that gets quoted is from the paper's text: GPT-3.5-Turbo's multi-document performance "can drop by more than 20%", and in the worst case its accuracy with 20 or 30 documents fell below its closed-book accuracy of 56.1%, meaning the retrieved documents made the answer worse than no documents at all. The oracle setting, where only the relevant document is supplied, scored 88.3%. The gap between those two is the cost of retrieving too much and placing it badly.

The limit is the date. The models tested were GPT-3.5-Turbo, Claude 1.3, MPT and LongChat, all from 2023. Nothing we read re-ran the protocol on 2026 long-context models, and we do not claim the effect persists at the same size. What the paper supports today is a design rule rather than a number: retrieve fewer, better-ranked passages, and put the most relevant first or last. The failure case it describes is a system that pads the context to the window limit on the theory that more is safer.

The newest taxonomy gives chunking its own stage. In "Classifying and Addressing the Diversity of Errors in RAG Systems", posted October 15, 2025 and published at EACL 2026, Leung and colleagues classify sixteen error types across four stages: chunking, retrieval, reranking and generation. The first three types are chunking errors, overchunking, underchunking and context mismatch, and the paper's mitigation advice is specific and honest: larger chunks reduce overchunking, smaller chunks reduce underchunking, small overlaps mitigate context mismatch, and these approaches "require careful tuning and are often insufficient on their own".

Read beside Anthropic's post, the two sources describe the same defect from the research side and the vendor side. A chunk is a fragment that has forgotten which document it came from, which section it sat in and what the surrounding paragraphs said. The vendor's answer is to write that context back in before indexing; the paper's answer is to tune the cut and accept that tuning alone will not finish the job. The example that catches teams is a contract split at a fixed token count, where the clause that answers the question starts in one chunk and ends in the next, so neither chunk matches the query on its own.

Barnett's last four failure points and Leung's last eight error types describe what goes wrong after retrieval has succeeded. Barnett's FP4, not extracted, is the model failing to pull the answer out of a context that contains it, which the paper says typically happens under noise or contradicting information. FP5, FP6 and FP7 are shape errors: wrong format, wrong specificity, and answers that are correct but incomplete, for which the paper's remedy is to ask the sub-questions separately.

Leung and colleagues split the generation stage finer. Their list runs from abstention failure, where the system answers when it should have declined, through fabricated content and parametric over-reliance, where the model answers from its training rather than the retrieved text, to misinterpretation, contextual misalignment, chronological inconsistency and numerical error. The last two are the ones a generic hallucination check misses: a date transposed from an older version of a document, or a figure copied with the wrong unit, is faithful to something in the context and still wrong.

The router below maps the symptom a user reports to the stage the research says to inspect first. It is a reading of the papers, not a diagnostic we have validated on any system.

The field's count of failure modes has grown with each paper, which can read as disagreement and is mostly resolution. Barnett counted seven from the outside, as a user sees them. Leung counted sixteen by the stage that produced them. The widest current list is a systematic taxonomy presented at the TrustNLP workshop in July 2026, whose abstract describes "33 failure modes across 7 pipeline stages — ingestion, representation, retrieval, generation, evaluation, deployment, and agentic orchestration", built from a structured review of 48 sources with each mode's evidence graded strong, moderate or limited.

The review's most useful sentence is about what is missing. It finds that "retrieval and generation failures are comparatively well-studied, while representation, evaluation, and agentic orchestration failures remain under-investigated". That matches the shape of this post: the numbers all live in retrieval and position, and the stages where agents chain retrievals together have taxonomy entries but little measurement. Our agentic RAG patterns guide describes those architectures; the evidence that they fail in specific, measured ways is thinner than for the single-hop case.

Two 2023 papers supply two of the most-cited metric frameworks. Ragas, by Es and colleagues, posted September 26, 2023, defines three reference-free scores: faithfulness, the idea that "the answer should be grounded in the given context"; answer relevance; and context relevance, that the retrieved context "should be focused, containing as little irrelevant information as possible". On its own human-annotated evaluation set, agreement with human judgment was 0.95 for faithfulness, 0.78 for answer relevance and 0.70 for context relevance. The retrieval-side check was the weakest.

ARES, by Saad-Falcon and colleagues, posted November 16, 2023, scores the same three dimensions with fine-tuned language-model judges calibrated against a small set of human labels, and its abstract claims effectiveness "using only a few hundred human annotations". Both frameworks share a limit the whole field carries: the judge is a language model, and Barnett found the automated check more pessimistic than a human rater. A failure rate from an LLM judge is a signal, not a count, until a human sample confirms it.

The failure case in measurement is a dashboard that reports a faithfulness score and nothing about retrieval. By Ragas's own numbers, that is the dimension where the judge agrees with humans most, and it measures only the final stage of a pipeline the research says fails at several. The production plan puts retrieval recall on the first dashboard for that reason.

Four findings run through the sources. Failures arise at several stages, not only in the model. Retrieval-stage misses are a large, measurable share of them. Position in the context changed accuracy substantially in the models that were tested. And the 2024 experience report recommends continuous evaluation on live questions over a one-time benchmark, with Ragas and ARES supplying the metrics to do it.

Four things are open. Whether the lost-in-the-middle effect persists at the same size in 2026 long-context models has not been re-measured in anything we read. The vendor's 35%, 49% and 67% gains are vendor-run on private data. LLM-judge metrics and human judgment diverge, and nobody has settled by how much on production traffic. And the three taxonomies, at seven, sixteen and thirty-three modes, have not been reconciled against one another; they are three maps of the same territory at different scales. If you need the territory surveyed on your own system rather than in the literature, that is what our AI transformation engagements do, and the audit starts at retrieval.