Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) is a family of methods that retrieves information from an external collection and conditions a generative model on that information when producing an output. It connects information retrieval with generation so that an answer can draw on material not contained, or not reliably accessible, in a model's parameters. The external collection may contain passages, records, code, images, or other retrievable units, but this article focuses on text retrieval for natural language processing.

Patrick Lewis and colleagues introduced the named RAG architecture in 2020. Their system paired a dense retriever over a Wikipedia index with a pretrained sequence-to-sequence generator and treated the retrieved document as a latent variable ^[1]. The term is now also used more broadly for modular applications that retrieve passages and place them in the input of a large language model, even when those applications do not use the original training objective or model architecture.

Retrieval can make a source collection easier to update, expose passages for inspection, and improve task performance in particular settings. It does not by itself make an answer correct, current, complete, cited, or secure. A RAG system can retrieve the wrong material, miss necessary evidence, retrieve contradictory or malicious content, ignore good evidence, or generate a claim that its cited passages do not support. Any benefit is therefore a property of a specific corpus, retriever, generator, task, and evaluation, not of the label "RAG" alone.

The shape of the field changed substantially after 2023. Two developments dominate. Context windows grew from a few thousand tokens to a million or more, prompting a sustained argument that models can simply read whole collections and that retrieval is a workaround for a limitation that no longer exists. At the same time, retrieval moved inside agent loops: instead of one retrieval before one generation, a model issues several searches, inspects what comes back, and decides whether to search again. Both changes are covered below. Neither has settled the question of when retrieval is worth its cost, and the published evidence on that question points in more than one direction.

Scope and terminology

A minimal RAG system has four functional parts:

1. a source collection whose items have identifiers and content;
2. an index or search mechanism that can select items for an input;
3. a context-assembly step that presents selected material to a generator; and
4. a generator that produces an output from the input and assembled context.

Many deployments add query rewriting, metadata filters, reranking, deduplication, citation construction, access control, caching, monitoring, or multiple rounds of retrieval. Those additions change the system's behavior, but they are not required by the broad definition.

The source collection is sometimes called non-parametric memory because its contents can be changed without directly changing the generator's learned weights. Knowledge encoded in those weights is called parametric memory. The distinction is useful but not absolute. A learned retriever has parameters; generated answers may mix retrieved information with parametric knowledge; and updating an index can require recomputing representations. RAG is therefore not equivalent to a vector database, a search engine, a prompt template, or a particular model.

Retrieval means selecting information in response to an input. Augmentation means supplying retrieved information to the computation that produces the output. Generation means constructing an output sequence rather than merely returning a stored passage. A retrieve-and-extract question-answering system and a search interface may be adjacent technologies, but they are not necessarily RAG under this definition.

Grounding and attribution are separate claims. An answer is grounded only to the extent that its claims follow from the stated evidence under an appropriate interpretation. An answer is attributed when it identifies sources or passages. A citation can be correctly formatted yet fail to support the nearby claim, and a well-supported answer can omit a citation. Freshness is also separate: retrieval can expose newer material only if the collection, index, filters, and timestamps are maintained correctly.

A further distinction has become important as systems grew more autonomous. In a pipeline sense, RAG is a fixed sequence executed once per request: retrieve, assemble, generate. In a capability sense, retrieval is one action available to a model that also plans, calls other tools, and decides how many times to act before answering. Surveys of agentic RAG describe this second arrangement as embedding autonomous agents into the retrieval pipeline so that retrieval strategy is chosen at run time rather than fixed in advance ^[49]. The two senses share components but differ in where control lives, what can be measured offline, and which failure modes appear. Statements about "RAG performance" are ambiguous unless they say which sense is meant.

Historical development

Retrieval-assisted language modeling predates the 2020 RAG paper. REALM trained a masked language model with a latent retriever used during pretraining, fine-tuning, and inference, showing that retrieval could be learned from the language-modeling objective ^[2]. Dense Passage Retrieval then demonstrated a dual-encoder retriever for open-domain question answering, representing questions and passages separately and ranking passages by vector similarity ^[3]. The original RAG model initialized its retriever from that work.

Another line retrieved examples at token-prediction time. The kNN-LM method built a datastore from a language model's hidden states and interpolated the model's next-token distribution with a nearest-neighbor distribution, without additional model training in the studied setup ^[4]. This is retrieval-augmented language modeling, but it does not retrieve human-readable passages for a prompt in the same way as a typical application-level RAG pipeline.

Research after 2020 explored different points of integration. Fusion-in-Decoder encoded each question-passage pair separately and let the decoder combine the encoded evidence, avoiding a single early concatenation of all passages ^[5]. RETRO incorporated retrieval into autoregressive language-model pretraining and retrieved neighboring chunks from a very large token database ^[6]. Atlas jointly trained a retriever and generator for few-shot learning and allowed the non-parametric memory to be updated ^[7]. These systems show that retrieval can enter pretraining, fine-tuning, or inference; their architectures are not interchangeable.

Evaluation work also began to couple outputs with provenance. KILT provided a shared Wikipedia snapshot and evaluated both a task output and the pages offered as evidence ^[8]. Later methods changed when and why retrieval occurs. Self-RAG trained a model to use reflection tokens for retrieval and for judgments about relevance, support, and utility ^[9]. FLARE triggered retrieval while generating long-form text, using anticipated future content and confidence in the studied method ^[10]. IRCoT interleaved retrieval with intermediate reasoning for multi-step questions ^[11]. HyDE generated a hypothetical document as a query representation and then retrieved real documents near its embedding ^[12]. In HyDE, the generated hypothetical document is not evidence; the retrieved corpus items remain the candidate evidence.

These developments produced a broad design space rather than a single standard architecture. A useful description of a RAG system should state what is retrieved, from where, at which stage, with what supervision, and how retrieved material affects generation.

From single-shot pipelines to search policies

The period from 2024 onward is better described by a change in control flow than by a change in components. Three shifts recur across the literature.

The first is structure in the index. Instead of a flat list of passages, systems build hierarchies of summaries, entity graphs, or per-chunk context annotations so that retrieval can return material at more than one level of abstraction ^[42][43][50].

The second is multiple retrieval steps under model control. Rather than one query derived from the user's input, the model produces a sequence of queries, reads results, and decides when it has enough ^[11][47][48].

The third is retrieval as one tool among several. Search sits alongside code execution, database queries, and file reading in a general action space, which makes the retrieval subsystem harder to isolate for measurement.

The table below places the main reference points in order. It is a map of published methods, not a ranking; several entries solve different problems and are not substitutes for each other.

Scroll sideways for more →

Architecture and operation

Corpus construction and indexing

The corpus determines what the system can retrieve. Corpus design includes source selection, licensing, extraction, normalization, segmentation, versioning, and retention. A source identifier should remain attached to every indexed unit so that an answer can be traced back to the exact item and version used. When permissions differ by user or tenant, authorization must be enforced before or during retrieval, not merely hidden in the user interface after retrieval.

Long documents are often divided into passages or chunks because retrieval and generation operate within finite computation and context-window limits. Boundaries affect meaning. Small units may isolate a precise fact but lose surrounding definitions, tables, or exceptions; large units preserve more context but consume more input space and may dilute the relevant part. Overlap can preserve material near a boundary while increasing duplication. Structural segmentation can respect sections, paragraphs, records, or code units, and semantic chunking methods place boundaries where the topic shifts rather than at fixed lengths. There is no evidence-based universal chunk length. The original RAG experiments used disjoint 100-word Wikipedia chunks, which was a choice for that corpus and experiment, not a general rule ^[1].

Two indexing strategies address the same underlying problem, which is that a chunk removed from its document loses the information needed to interpret it.

Chunk-level context annotation prepends a short generated description of each chunk's place in its source document before embedding and before building the lexical index. Anthropic published this approach as contextual retrieval in September 2024 and reported, on its own evaluation across codebases, fiction, and scientific papers, that contextual embeddings alone reduced the top-20 retrieval failure rate from 5.7 percent to 3.7 percent, that combining contextual embeddings with a contextual BM25 index reduced it to 2.9 percent, and that adding a reranking stage reduced it further to 1.9 percent ^[43]. These are the vendor's own numbers on the vendor's own evaluation set, using recall at 20 as the measure; they establish an ordering among the variants tested rather than a transferable effect size.

Hierarchical summarization builds a tree instead of a flat list. RAPTOR recursively embeds, clusters, and summarizes chunks from the bottom up, so that retrieval can draw on leaf passages or on summaries covering larger spans of a document. Its authors reported state-of-the-art results on question answering requiring multi-step reasoning, including a 20-point absolute accuracy improvement on the QuALITY benchmark when RAPTOR retrieval was paired with GPT-4 ^[42]. The cost is that summaries are model-generated: they are a derived artifact that can lose or distort content, and they must be rebuilt when sources change.

A lexical index represents terms and their statistics. BM25, for example, belongs to a probabilistic relevance framework and scores term matches using document and collection statistics ^[13]. A dense index stores learned embeddings and retrieves items whose vectors are close to a query vector. Dense Passage Retrieval is one influential dual-encoder design ^[3]. Hybrid search combines signals, commonly lexical and dense scores or ranks, to capture both exact terminology and semantic similarity. Learned sparse methods such as SPLADE sit between the two families, producing a sparse term-weighted representation from a neural model so that it can be served by an inverted index.

No retrieval family dominates every domain. BEIR evaluated lexical, sparse, dense, late-interaction, and reranking systems over heterogeneous zero-shot datasets. Performance varied substantially across datasets, and several dense systems generalized poorly to some domains despite strong results in narrower settings ^[14]. ColBERT instead retains multiple token-level representations and uses late interaction; ColBERTv2 studied compression and denoised supervision to reduce its storage footprint while preserving its retrieval approach ^[15]. These results support measuring retrieval on representative data rather than selecting a method by architecture name.

Exact search over every dense vector can be expensive. Approximate-nearest-neighbor indexes trade some recall for speed or memory. HNSW organizes proximity links in a multilayer graph and searches the graph approximately ^[16]. Libraries such as FAISS implement several such structures along with vector quantization. Index parameters, vector quantization, filtering, and update strategy can all change the candidates returned even when the same embedding model is used. Retrieval evaluation must therefore cover the deployed index, not only an offline similarity function.

Query processing and retrieval

The user's input is not always an effective search query. A pipeline may normalize it, add conversational context, extract entities, generate several queries, or decompose a multi-part question. Rewriting can improve recall, but it can also discard a constraint or introduce an assumption. Systems should retain the original request and test whether rewritten queries preserve its meaning.

A first-stage retriever usually returns a candidate set. Metadata filters can restrict dates, languages, source types, permissions, or document states. A reranker can then score the query and each candidate jointly, often at higher computation per candidate than a dual encoder. The number of retrieved candidates, the number passed to the generator, and any minimum-score rule are distinct controls. A fixed top-K rule always supplies K items even when none is useful unless the system also applies a threshold or an explicit no-evidence decision.

Multi-step questions may require evidence that does not share vocabulary with the original query. Iterative systems use an answer fragment or intermediate state to retrieve again. This can recover bridging evidence, as studied by IRCoT ^[11], but each step can also compound an early error or fill the context with redundant passages. Adaptive retrieval methods try to stop when evidence is sufficient, yet sufficiency itself must be evaluated.

Pipeline choices interact. An EMNLP 2024 study compared combinations of query classification, chunking, retrieval, reranking, repacking, summarization, and generation in its experimental settings and found performance-efficiency tradeoffs rather than a cost-free universal configuration ^[17]. Its recommendations remain conditional on the tested datasets and models. Deployment values for chunk size, overlap, candidate count, reranking depth, and context order should be selected against representative tasks, latency limits, and error costs.

Context assembly and generation

Context assembly decides which retrieved units the generator actually sees and how they are presented. Common operations include removing duplicates, grouping neighboring passages, preserving document boundaries, labeling sources, ordering evidence, and fitting the material into an input budget. A rank produced for retrieval relevance is not automatically the best order for generation. The system may need to keep a definition next to its qualifier or represent two conflicting sources separately.

Ordering has measurable effects. Researchers at NVIDIA proposed order-preserving RAG, which places selected chunks in their original document order rather than in score-descending order, and reported that answer quality under this arrangement rises and then falls as more chunks are added, tracing an inverted U. They identified operating points at which their method reached higher answer quality than supplying the whole context to a long-context model, using far fewer tokens ^[34]. A separate study of long inputs in RAG found the same non-monotonic pattern and attributed the decline to hard negatives, meaning retrieved passages that look relevant but are misleading; it proposed reordering retrieved passages as a training-free mitigation ^[39]. Both results argue against the intuition that more retrieved passages are always better, and both were obtained on specific models and datasets.

The generator receives the request plus the assembled context. Instructions can ask it to use only the supplied evidence, distinguish uncertainty, quote identifiers, or abstain when support is missing. Those instructions shape behavior but do not enforce a logical constraint. A model can ignore context, blend it with parametric memory, misunderstand a passage, or attach a citation to an unsupported sentence.

An evidence-aware output path preserves the mapping from output claims to source units. One approach lets the generator emit source identifiers; another aligns generated spans with passages after generation. Post-hoc citation matching can improve formatting but cannot turn an unsupported claim into a supported one. If the task requires exact extraction, calculation, or database semantics, a deterministic component may be more appropriate than asking a language model to reproduce the operation from prose.

Retrieval quality in practice

Most of a RAG system's answer quality is decided before the generator runs. If the evidence never enters the context, no prompt repairs the omission. This section covers the components that determine what reaches the generator.

Combining lexical and dense signals

Lexical and dense retrieval fail in different ways. Term matching handles exact identifiers, product codes, rare names, and negations that embeddings blur together, but it misses paraphrase. Dense retrieval handles paraphrase but can rank a topically similar passage above the one containing the specific string the user asked about. Combining them is common enough to be a default in production search stacks.

Score-level combination requires calibrating two scores that are not on the same scale. Rank-level combination avoids that problem. Reciprocal rank fusion, introduced by Cormack, Clarke, and Buettcher at SIGIR 2009, combines ranked lists using only the position of each document in each list, with a small constant to damp the influence of top ranks. In the original paper it outperformed Condorcet fusion and individual learning-to-rank methods on the tested collections ^[44]. Its persistence is partly a property of the method and partly a property of the problem: it needs no training data, no score normalization, and no knowledge of how the component systems work.

Fusion is not free of failure modes. A retriever that reliably returns nothing useful still contributes ranks, and rank-only fusion cannot distinguish a confident first result from a weak one. Systems that need an explicit "no relevant evidence" decision have to make it before or after fusion rather than expecting fusion to produce it.

Reranking

A first-stage retriever must score a whole collection, which limits it to representations that can be precomputed and compared cheaply. A reranker only scores a shortlist, so it can process the query and the candidate together. A cross-encoder does this by feeding the concatenated pair through a transformer and reading a relevance score, which lets attention operate across the query and passage jointly rather than comparing two independently produced vectors.

Reranking is one of the few pipeline additions with consistent published support. The EMNLP 2024 best-practices study found reranking useful across its tested configurations while noting the efficiency cost ^[17]. Anthropic's contextual retrieval evaluation reported that adding a reranking stage on top of an already-improved hybrid index reduced the top-20 failure rate from 2.9 percent to 1.9 percent, the single largest remaining gain in its sequence of changes ^[43]. HELMET, a long-context benchmark, includes reranking as one of its seven task categories, treating it as a distinct capability rather than a subcase of retrieval ^[37].

A common practitioner claim is that adding a reranker matters more than switching embedding models. The published evidence supports a weaker version of that statement: reranking reliably improves ranking quality on top of a given first stage, and the measured gains from reranking in the studies above are larger than the gaps typically reported between competent modern embedding models on the same data. That is not the same as a general ordering. A first stage with poor recall cannot be repaired by reranking, because a reranker can only reorder what it is given. The practical implication is sequencing rather than ranking of importance: measure first-stage recall at the depth you intend to rerank, fix recall if it is the binding constraint, then add reranking to convert recall into precision at the top of the list.

Rerankers add latency proportional to shortlist depth and cost proportional to the number of pairs scored. Reranking 100 candidates instead of 20 is a five-fold increase in reranker compute for a gain that has to be measured, not assumed.

Late interaction and multi-vector retrieval

Late-interaction models occupy a middle position. Instead of one vector per passage, they store one vector per token and compute relevance by summing, over query tokens, the maximum similarity to any passage token. ColBERT introduced the approach and ColBERTv2 addressed its storage cost through residual compression and denoised supervision ^[15]. The tradeoff is explicit: much larger indexes in exchange for finer-grained matching than a single pooled vector allows.

The approach extended to documents that are not primarily text. ColPali, published at ICLR 2025, embeds rendered page images with a vision-language model and applies the same late-interaction matching, retrieving pages directly rather than running optical character recognition, layout detection, and chunking first ^[45]. For collections of scanned reports, slide decks, and forms, this removes a fragile extraction stage, at the cost of an index that stores many vectors per page and a retrieval unit (the page) that may be coarser than the evidence actually needed.

Choosing and maintaining an embedding model

Embedding choice is usually made by consulting a leaderboard. MTEB and its multilingual successor MMTEB, published at ICLR 2025, aggregate hundreds of tasks across many languages, including instruction following, long-document retrieval, and code retrieval. One result worth noting is that scale did not determine the ranking: the authors reported that although models with billions of parameters achieved the best results on some language subsets and task categories, the best-performing publicly available model overall in their evaluation was a 560-million-parameter multilingual model ^[46]. Leaderboard position is also vulnerable to the same contamination problems described later in this article, since public benchmark data can enter embedding training sets.

Two operational considerations receive less attention than they deserve. First, an embedding model is a long-term commitment: changing it invalidates every stored vector, so migration means re-embedding the corpus and, during the transition, maintaining two indexes or accepting a period of degraded retrieval. Second, query and document vectors must come from compatible models. A silently upgraded query encoder against an unchanged document index produces retrieval that degrades without erroring.

What the evidence supports

Across these components, the reliable findings are narrow:

Scroll sideways for more →

The recurring pattern is that each of these choices was validated on particular corpora with particular models, and that the size of the reported effect is not portable even when its direction is. A representative evaluation set built from the deployment's own data settles more than any leaderboard.

Iterative and agentic retrieval

Why one retrieval step is often not enough

Single-shot retrieval assumes the user's request contains enough signal to find the evidence. Several common request types break that assumption. A comparison question needs evidence about two entities that no single query ranks together. A multi-hop question needs a bridging fact that is only expressible after the first fact is known. An ambiguous question needs disambiguation before retrieval is meaningful. A question about a document's overall argument needs coverage rather than the single closest passage.

IRCoT addressed part of this by interleaving retrieval with intermediate reasoning steps for multi-step questions ^[11], and FLARE addressed another part by triggering retrieval during long-form generation rather than only at the start ^[10]. Both changed when retrieval happens. The methods described below change who decides.

Self-assessment and correction

Self-RAG trained a model to emit reflection tokens marking whether retrieval is needed and whether a retrieved passage is relevant, supportive, and useful ^[9]. Corrective RAG took a different route: a lightweight retrieval evaluator scores the retrieved set and returns a confidence value, which selects among knowledge actions, and a decompose-then-recompose step strips irrelevant material from the passages that are kept. Its authors designed it to attach to existing RAG pipelines without retraining the generator and reported improvements across four datasets covering short- and long-form generation ^[47].

Both approaches make the same bet: that the system can estimate whether its evidence is adequate. That estimate is itself a model output and can be wrong in both directions. A system that over-trusts its own confidence answers from bad evidence; one that under-trusts it retrieves repeatedly and still abstains. The relevant measurement is not the average quality of the estimate but its behavior at the decision threshold the deployment actually uses. Related verification techniques such as chain-of-verification apply the same idea after generation, drafting an answer and then checking its claims, which shifts the cost from retrieval to verification without removing the need for a reliable judgment of sufficiency.

Learned search policies

A more recent line trains the search behavior itself rather than prompting for it. Search-R1 applies reinforcement learning so that a model learns to issue multiple search queries during step-by-step reasoning against a live retriever, using retrieved-token masking to stabilize training and an outcome-based reward. Across seven question-answering datasets its authors reported relative improvements over RAG baselines of 41 percent with a 7-billion-parameter Qwen2.5 model and 20 percent with the 3-billion-parameter version, under matched settings ^[48]. Several contemporaneous methods pursue the same idea with different reward formulations and retrieval environments, and a 2025 survey catalogues the design space of agents that plan, reflect, and use tools inside the retrieval loop ^[49].

Two cautions apply to this literature. Reported gains are measured against RAG baselines chosen by the authors, and a weak baseline inflates the difference. And outcome-based rewards optimize the final answer, which means a policy can learn to reach the right answer through retrieval trajectories that a human reviewer would not endorse, including trajectories that ignore retrieved evidence when parametric knowledge suffices.

Retrieval as a tool call

In production agent systems, search is typically exposed through tool use or function calling: the model emits a structured call, a runtime executes it, and the result returns as a message in the conversation. The Model Context Protocol is one interface standard for connecting such tools to models. Deep research systems are the most visible application, running many searches over minutes and returning a cited report.

This arrangement has practical consequences that are easy to miss.

Evaluation becomes harder. There is no single retrieval step to score. Recall at K is undefined when the number and content of queries are chosen at run time, so retrieval quality has to be assessed over whole trajectories, which are expensive to label.

Cost and latency become variable and hard to bound. A request may trigger one search or twenty. Budgets have to be enforced by the harness, since a model asked to be thorough has no reason to stop early.

Context management becomes the binding constraint. Each retrieval appends to a growing transcript, so a long trajectory can fill the window with intermediate results before reaching a conclusion. This is the practical origin of context engineering as a distinct concern: deciding what to keep, summarize, or discard between steps.

The security boundary moves. In a single-shot pipeline, retrieved text is inert input. In an agent loop, retrieved text sits in the same channel as instructions and is read by a model that can call tools. That difference is the subject of the security section below.

Graph and structured retrieval

The problem graphs address

Vector similarity answers questions whose evidence is localized in one or a few passages. It cannot answer a question whose evidence is distributed across an entire corpus. "What are the main themes in this dataset?" has no nearest neighbor, because no passage contains the answer. Microsoft Research framed this as query-focused summarization rather than retrieval and proposed GraphRAG to handle it: a language model extracts an entity knowledge graph from the source documents, communities of closely related entities are detected, summaries are pregenerated for every community, and a query is answered by generating partial responses from community summaries and then summarizing those. On global sensemaking questions over datasets in the million-token range, the authors reported substantial improvements over a conventional RAG baseline in the comprehensiveness and diversity of answers ^[50].

The distinction worth carrying forward is between local questions, whose answers live in specific passages, and global questions, whose answers require aggregating across the collection. These need different index structures, and a system tuned for one will be poor at the other.

Cost profile

GraphRAG's cost is concentrated in indexing. Extracting entities and relationships from every document and pregenerating summaries for every community requires many model calls before the first question is asked, and the whole structure must be rebuilt or incrementally maintained as sources change.

Microsoft Research later published LazyGraphRAG, which defers the expensive work to query time. Its stated claims are specific: Microsoft states that LazyGraphRAG's data indexing costs are identical to vector RAG and 0.1 percent of the costs of full GraphRAG, that a comparable configuration matches GraphRAG global search on answer quality for global queries at more than 700 times lower query cost, and that at 4 percent of the query cost of GraphRAG global search it outperformed all of the competing methods in the comparison ^[51]. Those figures come from the developing organization's own blog post of November 2024, evaluated on 5,590 Associated Press news articles with 100 synthetic queries scored by language-model comparison on comprehensiveness, diversity, and empowerment.

Microsoft subsequently released BenchmarkQED in June 2025, an open-source suite for generating queries across the local-to-global spectrum and scoring answers pairwise, and reported that in that evaluation LazyGraphRAG achieved higher win rates than vector RAG configured with a 1-million-token context window on comprehensiveness, diversity, and empowerment across query classes, while vector RAG retained a slight advantage on relevance for data-local queries ^[52]. The result is from the same organization that built both the method and the benchmark, which does not make it wrong but does mean it is not independent replication.

Contested evidence

Independent evaluation has been less favorable and more conditional than the original results suggest.

A systematic comparison by Han and colleagues built a unified protocol standardizing preprocessing, retrieval configuration, and generation settings so that RAG and GraphRAG could be compared on the same footing. The authors found distinct strengths for each rather than a winner, and reported complementarity on a multi-hop dataset where a subset of queries was answered only by GraphRAG and a comparable subset only by conventional RAG. On that basis they explored routing and integration strategies that combine both, which improved consistently over either alone ^[53].

A benchmark paper published in 2025 and revised in 2026 opens from the observation that recent studies report GraphRAG frequently underperforming vanilla RAG on real-world tasks, and constructs GraphRAG-Bench specifically to determine the conditions under which graph structure helps, evaluating the full pipeline from graph construction through retrieval to generation across fact retrieval, complex reasoning, contextual summarization, and creative generation ^[54].

The honest summary as of mid-2026 is that graph-structured retrieval has a clear motivating case, that its advantage on global sensemaking questions is reasonably well supported by the work that introduced it, that its advantage on ordinary factual question answering is not established and is frequently reversed, and that the cost claims for the cheaper variants come from their developers. A deployment considering it should first establish that its actual query mix contains global questions in meaningful volume, because if it does not, the graph is expensive index-building for queries that a hybrid retriever already answers.

Structured sources that are not graphs

Graphs are one form of structure. Many corpora also have tables, relational databases, ticketing systems, and APIs. For these, translating a request into a query language and executing it is usually better than embedding rows and retrieving them by similarity, because the operation required (filtering, joining, aggregating, counting) has exact semantics that similarity search does not reproduce. The LOFT benchmark found that long-context models still faced difficulty with the compositional reasoning required in SQL-like tasks even when they handled retrieval-style tasks well ^[27], which is a reason to keep the deterministic component rather than to expect the model to replace it. The general principle from the architecture section applies: when a task has exact semantics, implement the exact operation and use the model to formulate and explain it.

The long-context question

The most consequential open question about RAG is whether it is a workaround for short context windows. If a model can read a million tokens, the argument runs, then a retrieval pipeline is an unnecessary lossy filter that introduces its own failures. This section presents the argument, the counterargument, and the evidence on both, because the question is genuinely unsettled and the published results do not all point the same way.

The case for reading everything

The strongest version of this position is not rhetorical; it comes with measurements.

The LOFT benchmark, built around real-world tasks requiring context up to millions of tokens, found that long-context models could rival specialized retrieval and RAG systems despite never having been trained for those tasks. Its authors also emphasized advantages beyond accuracy: eliminating the need for specialized tool knowledge, avoiding cascading errors across pipeline stages, and allowing sophisticated prompting to apply across the whole system rather than to one stage of it ^[27].

A Google study compared RAG and long context directly across public datasets with three then-current models and concluded that when resourced sufficiently, long context consistently outperformed RAG on average, while adding that RAG's significantly lower cost remained a distinct advantage. Rather than declaring a winner, the authors proposed Self-Route, which asks the model whether the retrieved chunks suffice and escalates to full-context reading only when they do not, reporting comparable quality to long context at substantially reduced cost ^[33].

Vendor guidance points the same way. Google's Gemini API documentation, updated in June 2026, notes that limited context windows require strategies such as dropping old messages, summarizing, or using RAG with vector databases, and states that while these techniques remain valuable in specific scenarios, an extensive context window invites a more direct approach of providing all relevant information upfront. The same page acknowledges an inherent tradeoff between retrieving the right information and cost, and recommends context caching to reduce the expense of reusing large token sets ^[40].

The engineering argument that follows is real. A pipeline with chunking, embedding, indexing, filtering, reranking, and assembly has more places to fail, more parameters to tune, and more state to keep synchronized than a single call that reads the documents.

Advertised context is not usable context

The counterargument begins by questioning the premise. A model's advertised window is the number of tokens it will accept, not the number over which it maintains its short-context behavior. Four independent lines of evidence bear on this.

"Lost in the Middle" found that performance in multi-document question answering and key-value retrieval often varied with the position of the relevant information, and could be worse when that information sat in the middle of a long input rather than at either end ^[26].

RULER, presented at COLM 2024, extended needle-in-a-haystack testing with multiple needle types, multi-hop tracing, and aggregation. Evaluating 17 long-context models on 13 tasks, its authors found that despite near-perfect accuracy on the plain needle-in-a-haystack test, almost all models dropped substantially as context grew, and that although every model in the set claimed a window of 32K tokens or more, only half maintained satisfactory performance at 32K ^[35].

NoLiMa, presented at ICML 2025, removed the shortcut that makes needle tests easy. Its needles share minimal literal vocabulary with the question, so the model has to infer a latent association rather than match a string. Across 13 models claiming at least 128K tokens, 11 fell below half of their own sub-1K-token baselines at 32K; GPT-4o, one of the stronger exceptions, fell from 99.3 percent to 69.7 percent. The authors attributed the decline to the attention mechanism's difficulty in locating relevant information over long spans when literal matches are absent, and noted that reasoning-oriented models and chain-of-thought prompting did not rescue performance ^[36].

HELMET, presented at ICLR 2025, addressed why these disagreements persist. Studying 59 long-context models across seven application-centric task categories with controllable lengths up to 128K tokens, its authors found that synthetic tasks such as needle-in-a-haystack do not reliably predict downstream performance, that the different task categories correlate weakly with each other, and that while most models achieve perfect needle scores, open-weight models lag closed ones substantially when tasks require full-context reasoning or complex instruction following, with the gap widening as length increases ^[37]. Notably for this article, HELMET's authors recommended their RAG tasks as the fastest useful proxy during model development because those tasks predicted other downstream performance better than the synthetic ones.

Some advertised capacity also comes from context-extension methods applied to models not pretrained at that length. SelfExtend, an ICML 2024 spotlight, extends a model's usable window without fine-tuning by combining grouped attention over distant tokens with normal attention over nearby ones ^[41]. Techniques like self-extension make a longer window available; they do not by themselves establish that behavior at the new length matches behavior at the trained length.

The case that retrieval still earns its place

Beyond effective length, four arguments recur, and they are largely orthogonal to how good long-context models become.

Cost. Reading a large corpus per request is priced per token on every request. Caching reduces the cost of a repeated prefix, as Google's own documentation recommends ^[40], but caching helps when many requests share the same large context and helps little when each request concerns a different subset. Retrieval front-loads cost into indexing, which is paid once per document rather than once per query. The Google comparison that favored long context on quality named cost as RAG's distinct advantage and built its hybrid method around exactly that asymmetry ^[33].

Freshness and scale. A context window is a per-request budget, not a storage system. Corpora that exceed any window, or that change between requests, still require selection. Reindexing a changed document is cheaper than re-reading a collection on every request.

Attribution. Retrieval produces an explicit list of the units that were supplied, with identifiers and versions, before the model writes anything. Recovering the same trail from a model that read a million tokens means reconstructing which spans it actually used, which is a harder inference problem than recording a retrieval result. For regulated or high-consequence use, the auditable list is often the point.

Access control. Filtering by permission is a retrieval operation. If a system loads a whole corpus into context, the permission decision has already been made, in the direction of showing everything. Per-user filtering at retrieval time is a boundary that a long context window cannot express.

There is also direct evidence against the assumption that more context monotonically improves RAG. Databricks Mosaic Research ran more than 2,000 experiments across 13 open and closed models on four datasets and found that although retrieving more documents generally helped up to a point, most models degraded past a threshold that varied by model, with performance beginning to decline after roughly 32K tokens for Llama 3.1 405B and after roughly 64K for GPT-4-0125-preview in that study. The failure modes were model-specific rather than a uniform accuracy decay: Claude 3 Sonnet increasingly refused on copyright grounds as context grew, from 3.7 percent of cases at 16K to 49.5 percent at 64K, and DBRX increasingly summarized instead of answering, from 5.2 percent at 8K to 50.4 percent at 32K ^[38]. These are August 2024 measurements on models that have since been superseded, and the specific thresholds should not be carried forward, but the shape of the finding, that long-context failure is qualitative and model-specific rather than a smooth decline, has held up in the later work described above.

The order-preserving and hard-negative results discussed under context assembly cut the same way: adding retrieved passages improves answers up to a point and then degrades them, with the peak well short of the window's capacity ^[34][39].

Comparing the two honestly

Published comparisons frequently differ because they hold different things constant. A defensible comparison should fix the answer quality target, the corpus and its coverage, the per-request latency budget, the per-request and amortized cost, the freshness requirement, the attribution requirement, and the access-control boundary. Changing any one of these can reverse the conclusion. A study that measures accuracy alone on a static, small, public, permission-free corpus is not measuring the conditions under which most deployments choose retrieval.

Some of the disagreement is also a benchmark artifact. If a benchmark's questions can be answered from parametric memory, neither retrieval nor long context is being tested, a problem discussed in the evaluation section below ^[58]. And if a benchmark's evidence is short and lexically similar to the question, the long-context configuration is being tested on the easiest possible case, which is the specific criticism NoLiMa was constructed to make ^[36].

Where this stands as of mid-2026

The defensible position as of July 2026 is that the question resolved into a design tradeoff rather than a winner.

Long-context reading is the better choice for a bounded collection that fits comfortably inside the effective (not advertised) window, where the interactions between parts are hard for a retriever to anticipate, where the same context is reused across many requests so caching applies, and where attribution and per-user access control are not requirements.

Retrieval is the better choice when the corpus exceeds any window, changes frequently, requires per-user authorization, or must produce an auditable evidence trail, and when per-request cost matters at volume.

Hybrid designs are common and are what several of the papers on both sides actually propose. Self-Route escalates to full-context reading only when retrieved chunks appear insufficient ^[33]. Retrieving a document set and then giving the generator a long contiguous span, rather than scattered chunks, combines a retrieval filter with long-context reading. Order-preserving assembly is a hybrid in the same spirit ^[34].

What has not happened is the disappearance of retrieval. Larger windows removed the necessity of aggressive chunking for small collections and made naive pipelines harder to justify, but the cost, freshness, attribution, and authorization arguments do not depend on window size, and the effective-context literature shows that the advertised numbers overstate what is usable ^[35][36][37]. Any claim in either direction should be read with attention to what was held constant.

The original probabilistic model

The 2020 architecture formalized retrieval as a latent variable. For an input, a retriever assigns a probability to each passage, while a generator assigns token probabilities conditioned on the input, a passage, and earlier output tokens. In the formulas below, x is the input, y is the output, z is a passage, eta and theta denote retriever and generator parameters, and the set Z contains the top-K passages under the retriever.

In RAG-Sequence, one latent passage accounts for the whole output sequence:

$$p_{\mathrm{seq}}(y \mid x) \approx \sum_{z \in \mathcal{Z}_K(x)} p_{\eta}(z \mid x) \prod_{i=1}^{N} p_{\theta}(y_i \mid x,z,y_{<i})$$

In RAG-Token, the passage is marginalized separately for each output position:

$$p_{\mathrm{tok}}(y \mid x) \approx \prod_{i=1}^{N} \sum_{z \in \mathcal{Z}_K(x)} p_{\eta}(z \mid x) p_{\theta}(y_i \mid x,z,y_{<i})$$

The original implementation used DPR for retrieval, BART for generation, and a dense index of a December 2018 Wikipedia snapshot. It fine-tuned the query encoder and generator while keeping the passage encoder and document index fixed ^[1]. Consequently, "end-to-end" in that experiment did not mean that every corpus representation changed during each update.

Many systems now called RAG do not calculate either marginal. They retrieve text with a separate service, concatenate selected passages with an instruction, and call a generator without jointly training the retriever and generator. That modular pattern is still retrieval-augmented generation in the broad sense, but claims about the Lewis et al. likelihood, training behavior, or experimental results do not automatically transfer to it. The agentic systems described above depart even further: a model that decides at run time how many searches to issue is not marginalizing over a fixed candidate set at all, and the probabilistic formulation offers no account of its behavior.

Training, updating, and inference

RAG components can be trained separately or jointly. A retriever may learn from query-passage relevance pairs, question-answer supervision, clicks, or synthetic examples. A generator may be pretrained independently and used through prompting, fine-tuned to use retrieved evidence, or trained jointly with a retriever. REALM, the original RAG model, RETRO, and Atlas illustrate different training objectives and update boundaries ^[2][1][6][7]. Reinforcement-learning methods add a fourth possibility: training neither the retriever nor the generator's language modeling but the policy that decides when and what to search ^[48].

Updating the corpus is not the same as updating the model. A source change may require re-extraction, re-chunking, re-embedding, index insertion or deletion, cache invalidation, and a new version identifier. If old vectors or cached answers remain active, the system can continue returning superseded material. Conversely, replacing a source collection does not erase conflicting parametric knowledge in the generator.

Derived index structures add their own update burden. A hierarchical summary tree, a chunk-context annotation, or an entity graph is generated from the source, so a source change invalidates the derived artifact as well as the raw chunk ^[42][43][50]. Systems with such structures need an explicit story for incremental maintenance, since full rebuilds are exactly the cost that motivated the lazier variants ^[51].

Retrievers also drift. A new embedding model may make stored vectors incompatible with new query vectors. A changed chunking policy alters both the retrieval unit and the citation target. A corpus snapshot may be internally consistent but stale. Reproducible evaluation records the corpus version, index build, retriever version, generator version, prompt or decoding configuration, and evaluation set.

At inference time, a well-specified system handles at least three states: useful evidence was found, evidence was found but conflicts, or no adequate evidence was found. Always answering can hide the last two states. Abstention or escalation can reduce unsupported answers, but it should be evaluated for both false refusals and failures to refuse.

Evaluation

RAG evaluation should separate the stages that can fail. End-to-end answer accuracy alone cannot show whether an error came from the corpus, retrieval, context assembly, or generation. Component diagnostics also prevent a generator's parametric knowledge from masking a retriever that did not find the evidence.

Retrieval evaluation

Retrieval evaluation requires queries, a defined corpus snapshot, and relevance judgments. Depending on the task, measures may include recall at K, precision at K, mean reciprocal rank, average precision, or normalized discounted cumulative gain. The metric should match the downstream need. If an answer requires two passages, retrieving only one may count as partial recall but still make the answer impossible. If the corpus has incomplete relevance labels, an apparently false positive may be an unjudged relevant item.

Retrieval quality includes more than topical similarity. Evidence may be topically relevant but outdated, unauthorized, contradicted, or insufficient for the requested claim. Evaluation can stratify by domain, language, date, document type, query ambiguity, multi-hop depth, and the presence of an answer in the corpus. BEIR's cross-domain results illustrate why an in-domain retrieval score should not be treated as a generalization guarantee ^[14], and MTEB and MMTEB show that model rankings shift by language and task category ^[46].

Agentic systems complicate this. When the model chooses its own queries at run time, there is no fixed query set to judge against, so retrieval quality has to be assessed over trajectories: did the system eventually surface the necessary evidence, how many steps and how much context did that take, and did it stop appropriately.

Generation and evidence evaluation

Generation evaluation can measure task correctness, faithfulness to provided evidence, relevance to the request, completeness, uncertainty handling, citation quality, and style. These dimensions can conflict. A concise answer may be relevant but omit a qualifier; a faithful answer can repeat an error in the source; a factually correct answer can be unsupported by the retrieved context because the generator supplied it from parametric memory.

RAGAS proposed reference-free measures for aspects of retrieved context, faithfulness, and answer quality ^[18]. ARES uses learned judges for context relevance, answer faithfulness, and answer relevance, combined with a small human-labeled set and prediction-powered inference ^[19]. These tools can accelerate experiments, but their scores depend on judge behavior, synthetic data, prompts, and task assumptions. They should be calibrated against human judgments and consequential errors in the target use.

Citation evaluation needs at least two questions: does each citation support the claim it is attached to, and are claims that need support actually cited? The ALCE benchmark evaluates long-form generation along correctness, citation quality, and fluency, and separates citation correctness from citation completeness ^[20]. Provenance-aware benchmarks such as KILT additionally test whether the system identified source pages, not just whether it produced a target string ^[8].

RAGTruth assembled nearly 18,000 manually annotated responses across question answering, data-to-text generation, and summarization to study hallucinations in RAG outputs ^[21]. Its existence reflects a basic evaluation fact: providing retrieved context does not make every generated statement entailed by that context. Human review remains important where a wrong synthesis, omitted exception, or misplaced citation has high cost.

Shared tasks and independent assessment

Framework-based evaluation is convenient but each team runs it on its own data with its own judges. Shared tasks with independent human assessment provide a different kind of evidence.

NIST ran a RAG track at TREC in 2024 and again in 2025. The 2025 edition introduced long, multi-sentence narrative queries intended to reflect search tasks that require reasoning rather than lookup, and it separated retrieval, retrieval-augmented generation, augmented generation, and automatic relevance judgment into distinct tasks. Participation spanned 46 runs from 12 groups on retrieval, 51 runs from 16 groups on the RAG task, 25 runs from 9 groups on augmented generation, and 36 runs from 5 groups on relevance judgment ^[55].

Two findings from that overview matter for anyone building an evaluation. First, support quality varied enormously across submitted systems: under manual assessment, weighted support precision and recall ranged from 0.03 to 0.95 depending on the system. Whether generated statements are actually backed by the cited passages is therefore a real differentiator between systems, not a solved property of the RAG pattern. Second, automated relevance judgment remained difficult, with the best runs reaching agreement fractions of 0.30 to 0.34 against human assessors, while model-based support assessment aligned well with human assessment at the run level ^[55]. The pattern is that automatic judges rank systems roughly correctly while disagreeing with humans on individual items, which is adequate for comparing configurations and inadequate for deciding whether one particular answer is grounded.

Vendor-published benchmark suites are a third category. Microsoft's BenchmarkQED generates queries across the local-to-global spectrum and evaluates answers pairwise with a language-model judge ^[52]. Such suites are useful and openly released, but results published by the organization that also built the system under test are not independent evidence.

Judging groundedness with language models

Most RAG evaluation now uses an LLM as a judge to decide whether an answer is supported by its context, because manual annotation does not scale. This introduces a circularity: the same class of model that produced the answer is asked to decide whether the answer follows from the evidence. Failure modes shared between generator and judge are invisible to the evaluation.

GroUSE, published at COLING 2025, was built to evaluate the evaluators. Its authors found that existing automated RAG evaluation frameworks overlook important failure modes even when GPT-4 is used as the judge, and that correlation with GPT-4's judgments is an incomplete proxy for a judge model's practical performance: open-weight judges that correlated strongly with GPT-4 did not generalize to the paper's criteria. Fine-tuning Llama-3 on GPT-4's reasoning traces improved both correlation and calibration ^[56].

Self-preference is a related and separately documented problem. Panickssery, Bowman, and Feng found that language-model evaluators score their own outputs higher than others' outputs that human annotators rate as equal in quality, that models can distinguish their own generations from those of other models and humans at non-trivial accuracy, and that the strength of self-preference correlated linearly with self-recognition ability in their fine-tuning experiments ^[57]. In a RAG evaluation this bias operates on both sides at once when the same model family generates answers and grades groundedness.

Practical mitigations follow from these results rather than solving them. Use a judge from a different model family than the generator. Score claims individually against specific passages rather than assigning one overall faithfulness number, since a single averaged score hides which sentence is unsupported. Maintain a human-labeled calibration set and re-measure judge agreement whenever the judge model or prompt changes. And treat judge scores as a way to rank configurations, which the TREC results indicate they do reasonably well, rather than as a verdict on an individual answer, which they do less well ^[55].

Contamination and benchmark leakage

RAG benchmarks have a failure mode that other benchmarks do not: a question that the model can answer from parametric memory tests nothing about retrieval. If a benchmark's questions are drawn from widely-mirrored public sources, a modern model has likely seen the answers during pretraining, and a system with a broken retriever can still score well.

This form of contamination worsens with time through benchmark aging. As benchmarks are reused, their contents are absorbed into training corpora and data-curation pipelines, so a benchmark that genuinely tested retrieval when published gradually stops doing so. Work published in 2026 addresses this by generating benchmark instances designed not to be answerable without retrieval, extracting a reasoning graph from question-context pairs and producing new examples through type-constrained entity replacement, with a verification step that excludes instances answerable from parametric knowledge alone ^[58].

Three checks are worth running on any RAG evaluation set. Measure the no-retrieval baseline, meaning the score with the retriever disabled entirely; whatever that baseline achieves is not evidence about retrieval. Include questions whose answers are absent from the corpus, so that abstention is measurable. And prefer private or freshly-generated data for decisions that matter, reserving public benchmarks for comparison against published numbers.

Operational evaluation

Operational measures include latency at each stage, index size, update delay, retrieval and generation cost, cache behavior, failure recovery, access-control correctness, and observability. Quality should be measured under the same context budgets and time limits used in production. An expensive reranker may improve a benchmark yet miss a latency objective; a cache may reduce latency while serving stale or unauthorized content.

Agentic retrieval makes these measures distributional rather than fixed. When the number of retrieval steps is chosen at run time, average latency and cost are less informative than their tail: the request that issues twenty searches determines the timeout budget and the worst-case bill. Step limits, per-request token budgets, and enforced stopping conditions belong in the harness.

Evaluation sets should include answerable and unanswerable requests, ambiguous queries, conflicting sources, outdated documents, malicious content, permission boundaries, and corpus changes. Online monitoring should not rely only on user ratings because users may not detect a plausible unsupported answer. Logged source identifiers and versions make later incident analysis possible, subject to privacy and retention constraints.

Failure modes and limitations

RAG introduces a pipeline of dependent components. It can improve a system only when the external information and the mechanism for using it are good enough for the task.

Retrieval and corpus failures

A coverage failure occurs when the necessary information is absent from the corpus. A retrieval miss occurs when it is present but not selected. A granularity failure occurs when segmentation separates evidence from a definition, header, table, exception, or neighboring passage needed to interpret it. A ranking failure places useful evidence below the context cutoff. Metadata errors can silently filter out the right source or admit a source the user is not permitted to see.

Retrieved material can also be low quality. Experiments on retrieval noise show that inappropriate passages can reduce answer quality in tested models ^[22]. Relevance is not truth: counterfactual-noise experiments found that relevant-looking but conflicting passages could mislead studied retrieval-augmented models ^[23]. Systems that search changing or user-contributed corpora need source validation, versioning, and conflict handling rather than assuming that high similarity means authority.

One widely-cited result in this area illustrates why individual findings should be treated cautiously. Cuconasu and colleagues reported at SIGIR 2024 that adding random or irrelevant documents to a RAG input could improve question-answering accuracy, a counterintuitive result that circulated as the "power of noise" ^[59]. A reproducibility study presented at SIGIR 2026 re-ran the experiments and reported a split verdict: the effect held under the original setup, which used earlier-generation quantized models with a restrictive extraction-style prompt and a strict decoding limit, but it appeared, weakened, or disappeared under small changes to prompt formulation and decoding limits, and error analysis traced substantial contributions to truncation and malformed generations. The authors concluded that the original effect could not be robustly confirmed as a general benefit of noisy retrieval ^[60]. The lesson generalizes past this one result: many published RAG findings are measurements of a particular inference configuration, and the configuration is often not the thing the paper claims to be studying.

The correct behavior when retrieval fails is itself an evaluation target. NoMIRACL contains relevant and non-relevant passage settings across 18 languages and tests both using relevant evidence and avoiding answers based on irrelevant evidence ^[24]. A system that refuses every difficult request avoids some hallucinations but has poor usefulness; a system that always answers has the opposite failure. Thresholds must be chosen for the application's error costs.

Generation and attribution failures

The generator may ignore a relevant passage, overgeneralize from it, combine incompatible sources, copy an error, or introduce unsupported details. Retrieval augmentation reduced hallucination in specific conversational experiments ^[25], but that result does not establish a general guarantee. The outcome depends on the source collection, retrieval quality, model, prompt, and task.

A citation is not proof of entailment. Citation-generation systems can attach a plausible source that discusses the topic without supporting the exact claim. They can also cite one passage for a sentence containing several claims when only one is supported. Verification should operate at the claim level and distinguish direct support, contradiction, missing evidence, and source-quality concerns. The TREC 2025 support measurements, spanning weighted precision and recall from 0.03 to 0.95 across submitted systems, indicate how wide the practical range is ^[55].

RAG cannot by itself resolve normative or interpretive questions. If sources disagree about policy, diagnosis, or causation, a generator should not silently collapse the disagreement into one confident answer. Source authority also depends on the question: a primary experiment can establish what its authors did, while a standards body may be authoritative about its own standard but not about an empirical effect outside its evidence.

Failures specific to multi-step retrieval

Agentic retrieval adds failure modes that single-shot pipelines do not have.

An early wrong turn propagates: a bad first query produces bad results, which shape the second query, and the trajectory never recovers. Loops occur when a model reissues variations of a failed query rather than concluding that the evidence is absent. Context saturation occurs when accumulated intermediate results crowd out the evidence needed for the final answer. Premature stopping occurs when a model judges partial evidence sufficient. And the cost of each of these is paid in latency and tokens before the user sees anything.

These are the flip side of moving control into the model. Bounding them requires limits imposed from outside the model: maximum steps, token budgets, deduplication across steps, and explicit termination criteria.

Security and governance

Indirect prompt injection through retrieved content

Retrieved content is an input channel, not a trusted instruction channel. Indirect prompt injection research demonstrated that attacker-controlled external content can manipulate LLM-integrated applications when that content is processed as part of a model's input ^[28]. In RAG, a malicious passage may tell the model to ignore system instructions, reveal data, call a tool, or produce attacker-chosen text. Delimiters and instructions can help a model distinguish data from commands, but they are not a complete security boundary.

This is the central RAG security problem, and it is not hypothetical. EchoLeak, tracked as CVE-2025-32711, was an indirect prompt injection against Microsoft 365 Copilot in which a single crafted email placed instructions into content that the assistant would later retrieve, enabling data exfiltration without the user clicking anything. Published analysis describes an exploit chain that evaded the product's cross-prompt injection classifier, circumvented link redaction, and used automatically fetched images as the exfiltration path; the work was presented at the AAAI Fall Symposium Series in 2025 ^[61]. Two properties of that case generalize. The attacker needed only the ability to put text where the system would index it, which in an enterprise assistant means anyone who can send email or share a document. And the payload was inert until retrieval brought it into a privileged context, so scanning at ingestion time and scanning at query time are different controls with different coverage.

The risk grows when retrieval sits inside an agent loop. A single-shot pipeline that concatenates passages and returns text gives an injected instruction limited reach. An agent that can call tools after reading retrieved content gives it the agent's full permissions.

Corpus integrity and poisoning

Corpus integrity is another attack surface. PoisonedRAG demonstrated knowledge-corruption attacks in which a small number of crafted texts inserted into a large corpus caused tested systems to produce attacker-chosen answers for target questions ^[29]. The reported attack rates belong to the paper's threat models and experiments; they should not be generalized to every system. The broader conclusion is that write access, ingestion, ranking, and source trust need security controls.

ConfusedPilot examined the same class of problem as a confused-deputy risk in enterprise deployments, describing an attack that embeds malicious text so that it corrupts generated responses, a second that leaks data by exploiting caching during retrieval, and the way both can propagate misinformation through an organization once a corrupted answer is itself written back into a shared document ^[62]. That last dynamic is specific to RAG over collaborative corpora: an answer saved into a wiki or a ticket becomes a retrievable source, and an error can persist after the original poisoned document is removed.

Data poisoning defenses for RAG are ordinary content-supply-chain controls rather than model controls: authenticated ingestion, provenance metadata on every indexed unit, review for sources with open write access, the ability to quarantine or remove a source and invalidate everything derived from it, and monitoring for anomalous retrieval patterns.

Access control and data leakage

OWASP's guidance on vector and embedding weaknesses highlights cross-context leakage, access-control failures, poisoning, and inadequate data validation ^[31]. Authorization filters must be applied to every retrieval path, including caches and fallback search. Tenant identifiers should not be treated as the only defense if the underlying index can return another tenant's content.

Two properties of this problem deserve emphasis.

First, permissions do not travel with a chunk by default. Access rules live in the source system; the extracted chunk in the vector store is just text and a vector unless permission metadata is deliberately copied across and kept synchronized. Revoking access in the source system does not revoke it in the index, and a shared index that crosses permission boundaries turns retrieval into a disclosure channel. Enforcement belongs at query time, applied as a filter to the retrieval itself rather than as a check after candidates are returned.

Second, an embedding is not anonymization. Morris and colleagues showed at EMNLP 2023 that dense text embeddings can be inverted: a multi-step method that iteratively corrects and re-embeds candidate text recovered 92 percent of 32-token inputs exactly and recovered full names from a dataset of clinical notes ^[63]. A vector store therefore requires the same access controls, encryption, and retention policy as the documents it was built from. Related work on membership inference shows that even without reconstruction, an adversary may be able to determine whether a particular document is present in an index, which is itself disclosure in some settings.

Governance

Risk management spans the full lifecycle. NIST's Generative AI Profile recommends attention to data and content provenance, third-party components, pre-deployment testing, monitoring, incident disclosure, and the limits of current measurement ^[30]. For RAG, this translates into documented source policy, authenticated ingestion, least-privilege retrieval, versioned indexes, tests for data leakage and conflicting evidence, output review proportional to consequence, and a way to remove or quarantine a source.

OWASP separately notes that RAG does not fully mitigate prompt injection ^[32]. Defenses are layered: restrict source and tool permissions, isolate untrusted content, validate retrieved metadata, minimize context, constrain tool calls outside the language model, monitor anomalous retrievals, and require human confirmation for irreversible or high-impact actions. None of these controls proves that a deployment is secure. A security claim must specify the attacker, assets, access, system boundaries, and tested controls.

Retrieval-augmented generation is best understood as a configurable evidence pipeline. Its value comes from making external information available at generation time and, when designed explicitly, preserving a trail back to sources. Its weaknesses come from the same dependency: the output inherits limitations from corpus construction, retrieval, context assembly, model behavior, and governance. Larger context windows and more autonomous agents changed which of those stages carries the most risk, moving it from segmentation and ranking toward context management, trajectory control, and the security of an input channel that a model can act on. They did not remove the underlying requirement. Reliable use still means measuring each stage and preserving the option to say that adequate evidence was not found.

References

1. ^Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, and others. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." In *Advances in Neural Information Processing Systems 33*, 2020. proceedings.neurips.cc/...6b493230-Abstract
2. ^Guu, Kelvin, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. "REALM: Retrieval-Augmented Language Model Pre-Training." In *Proceedings of the 37th International Conference on Machine Learning*, 2020. proceedings.mlr.press/...guu20a
3. ^Karpukhin, Vladimir, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. "Dense Passage Retrieval for Open-Domain Question Answering." In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing*, 2020. aclanthology.org/2020.emnlp-main.550
4. ^Khandelwal, Urvashi, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. "Generalization through Memorization: Nearest Neighbor Language Models." In *International Conference on Learning Representations*, 2020. openreview.net/forum
5. ^Izacard, Gautier, and Edouard Grave. "Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering." In *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics*, 2021. aclanthology.org/2021.eacl-main.74
6. ^Borgeaud, Sebastian, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, and others. "Improving Language Models by Retrieving from Trillions of Tokens." In *Proceedings of the 39th International Conference on Machine Learning*, 2022. proceedings.mlr.press/...borgeaud22a
7. ^Izacard, Gautier, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, and others. "Atlas: Few-shot Learning with Retrieval Augmented Language Models." *Journal of Machine Learning Research* 24, no. 251 (2023): 1-43. jmlr.org/...23-0037
8. ^Petroni, Fabio, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, and others. "KILT: a Benchmark for Knowledge Intensive Language Tasks." In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics*, 2021. aclanthology.org/2021.naacl-main.200
9. ^Asai, Akari, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection." In *International Conference on Learning Representations*, 2024. openreview.net/forum
10. ^Jiang, Zhengbao, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. "Active Retrieval Augmented Generation." In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 2023. aclanthology.org/2023.emnlp-main.495
11. ^Trivedi, Harsh, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. "Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions." In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics*, 2023. aclanthology.org/2023.acl-long.557
12. ^Gao, Luyu, Xueguang Ma, Jimmy Lin, and Jamie Callan. "Precise Zero-Shot Dense Retrieval without Relevance Labels." In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics*, 2023. aclanthology.org/2023.acl-long.99
13. ^Robertson, Stephen, and Hugo Zaragoza. "The Probabilistic Relevance Framework: BM25 and Beyond." *Foundations and Trends in Information Retrieval* 3, no. 4 (2009): 333-389. doi.org/...1500000019
14. ^Thakur, Nandan, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models." *Proceedings of the NeurIPS Track on Datasets and Benchmarks* 1, 2021. datasets-benchmarks-proceedings.neurips.cc/...t-round2
15. ^Santhanam, Keshav, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction." In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics*, 2022. aclanthology.org/2022.naacl-main.272
16. ^Malkov, Yu A., and D. A. Yashunin. "Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs." *IEEE Transactions on Pattern Analysis and Machine Intelligence* 42, no. 4 (2020): 824-836. doi.org/...TPAMI.2018.2889473
17. ^Wang, Xiaohua, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, and others. "Searching for Best Practices in Retrieval-Augmented Generation." In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, 2024. aclanthology.org/2024.emnlp-main.981
18. ^Es, Shahul, Jithin James, Luis Espinosa Anke, and Steven Schockaert. "RAGAs: Automated Evaluation of Retrieval Augmented Generation." In *Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations*, 2024. aclanthology.org/2024.eacl-demo.16
19. ^Saad-Falcon, Jon, Omar Khattab, Christopher Potts, and Matei Zaharia. "ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems." In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics*, 2024. aclanthology.org/2024.naacl-long.20
20. ^Gao, Tianyu, Howard Yen, Jiatong Yu, and Danqi Chen. "Enabling Large Language Models to Generate Text with Citations." In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 2023. aclanthology.org/2023.emnlp-main.398
21. ^Niu, Cheng, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. "RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models." In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics*, 2024. aclanthology.org/2024.acl-long.585
22. ^Fang, Feiteng, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. "Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training." In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics*, 2024. aclanthology.org/2024.acl-long.540
23. ^Hong, Giwon, Jeonghwan Kim, Junmo Kang, Sung-Hyon Myaeng, and Joyce Jiyoung Whang. "Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise." In *Findings of the Association for Computational Linguistics: NAACL 2024*, 2024. aclanthology.org/2024.findings-naacl.159
24. ^Thakur, Nandan, Luiz Bonifacio, Crystina Zhang, Odunayo Ogundepo, Ehsan Kamalloo, and others. "Knowing When You Don't Know: A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation." In *Findings of the Association for Computational Linguistics: EMNLP 2024*, 2024. aclanthology.org/2024.findings-emnlp.730
25. ^Shuster, Kurt, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. "Retrieval Augmentation Reduces Hallucination in Conversation." In *Findings of the Association for Computational Linguistics: EMNLP 2021*, 2021. aclanthology.org/2021.findings-emnlp.320
26. ^Liu, Nelson F., Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. "Lost in the Middle: How Language Models Use Long Contexts." *Transactions of the Association for Computational Linguistics* 12 (2024): 157-173. aclanthology.org/2024.tacl-1.9
27. ^Lee, Jinhyuk, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, and others. "Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" arXiv, 2024. arxiv.org/...2406.13121
28. ^Greshake, Kai, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection." arXiv, 2023. arxiv.org/...2302.12173
29. ^Zou, Wei, Runpeng Geng, Binghui Wang, and Jinyuan Jia. "PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models." In *34th USENIX Security Symposium*, 2025. usenix.org/...zou-poisonedrag
30. ^Autio, Chloe, Reva Schwartz, Jesse Dunietz, Shomik Jain, Martin Stanley, and others. "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile." NIST AI 600-1, 2024. nist.gov/...ork-generative-artificial-intelligence
31. ^OWASP GenAI Security Project. "LLM08:2025 Vector and Embedding Weaknesses." 2025. genai.owasp.org/...vector-and-embedding-weaknesses
32. ^OWASP GenAI Security Project. "LLM01:2025 Prompt Injection." 2025. genai.owasp.org/...llm01-prompt-injection
33. ^Li, Zhuowan, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. "Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach." In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track*, 2024. aclanthology.org/2024.emnlp-industry.66
34. ^Yu, Tan, Anbang Xu, and Rama Akkiraju. "In Defense of RAG in the Era of Long-Context Language Models." arXiv, September 2024. arxiv.org/...2409.01666
35. ^Hsieh, Cheng-Ping, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. "RULER: What's the Real Context Size of Your Long-Context Language Models?" In *Conference on Language Modeling (COLM)*, 2024. arxiv.org/...2404.06654
36. ^Modarressi, Ali, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schütze. "NoLiMa: Long-Context Evaluation Beyond Literal Matching." In *Proceedings of the 42nd International Conference on Machine Learning*, 2025. arxiv.org/...2502.05167
37. ^Yen, Howard, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. "HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly." In *International Conference on Learning Representations*, 2025. arxiv.org/...2410.02694
38. ^Databricks Mosaic Research. "Long Context RAG Performance of LLMs." Databricks blog, August 12, 2024. databricks.com/...long-context-rag-performance-llms
39. ^Jin, Bowen, Jinsung Yoon, Jiawei Han, and Sercan O. Arik. "Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG." arXiv, October 2024. arxiv.org/...2410.05983
40. ^Google. "Long context." Gemini API documentation, last updated June 22, 2026. ai.google.dev/...long-context
41. ^Jin, Hongye, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. "LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning." In *Proceedings of the 41st International Conference on Machine Learning*, 2024. arxiv.org/...2401.01325
42. ^Sarthi, Parth, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. "RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval." arXiv, January 2024. arxiv.org/...2401.18059
43. ^Anthropic. "Introducing Contextual Retrieval." September 19, 2024. anthropic.com/...contextual-retrieval
44. ^Cormack, Gordon V., Charles L. A. Clarke, and Stefan Buettcher. "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods." In *Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval*, 758-759, 2009. doi.org/...1571941.1572114
45. ^Faysse, Manuel, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. "ColPali: Efficient Document Retrieval with Vision Language Models." In *International Conference on Learning Representations*, 2025. arxiv.org/...2407.01449
46. ^Enevoldsen, Kenneth, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, and others. "MMTEB: Massive Multilingual Text Embedding Benchmark." In *International Conference on Learning Representations*, 2025. arxiv.org/...2502.13595
47. ^Yan, Shi-Qi, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. "Corrective Retrieval Augmented Generation." arXiv, January 2024. arxiv.org/...2401.15884
48. ^Jin, Bowen, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. "Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning." arXiv, March 2025. arxiv.org/...2503.09516
49. ^Singh, Aditi, Abul Ehtesham, Saket Kumar, Tala Talaei Khoei, and Athanasios V. Vasilakos. "Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG." arXiv, January 2025 (revised April 2026). arxiv.org/...2501.09136
50. ^Edge, Darren, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. "From Local to Global: A Graph RAG Approach to Query-Focused Summarization." arXiv, April 2024 (revised February 2025). arxiv.org/...2404.16130
51. ^Microsoft Research. "LazyGraphRAG: Setting a New Standard for Quality and Cost." November 25, 2024. microsoft.com/...new-standard-for-quality-and-cost
52. ^Microsoft Research. "BenchmarkQED: Automated Benchmarking of RAG Systems." June 5, 2025. microsoft.com/...mated-benchmarking-of-rag-systems
53. ^Han, Haoyu, Li Ma, Yu Wang, Harry Shomer, Yongjia Lei, Zhisheng Qi, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, Charu C. Aggarwal, and Jiliang Tang. "RAG vs. GraphRAG: A Systematic Evaluation and Key Insights." arXiv, February 2025 (revised March 2026). arxiv.org/...2502.11371
54. ^Xiang, Zhishang, Chuanjie Wu, Qinggang Zhang, Shengyuan Chen, Zijin Hong, Xiao Huang, and Jinsong Su. "When to Use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation." arXiv, June 2025 (revised February 2026). arxiv.org/...2506.05690
55. ^Upadhyay, Shivani, Nandan Thakur, Ronak Pradeep, Nick Craswell, Daniel Campos, and Jimmy Lin. "Overview of the TREC 2025 Retrieval Augmented Generation (RAG) Track." arXiv, March 2026. arxiv.org/...2603.09891
56. ^Muller, Sacha, António Loison, Bilel Omrani, and Gautier Viaud. "GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering." In *Proceedings of the 31st International Conference on Computational Linguistics*, 2025. arxiv.org/...2409.06595
57. ^Panickssery, Arjun, Samuel R. Bowman, and Shi Feng. "LLM Evaluators Recognize and Favor Their Own Generations." arXiv, April 2024. arxiv.org/...2404.13076
58. ^Liu, Jiayi, Jiaxing Zhang, Bowen Jin, and Jennifer Neville. "Generating Leakage-Free Benchmarks for Robust RAG Evaluation." arXiv, May 2026. arxiv.org/...2605.08838
59. ^Cuconasu, Florin, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. "The Power of Noise: Redefining Retrieval for RAG Systems." In *Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval*, 2024. doi.org/...3626772.3657834
60. ^Mazuryk, Michał, Fleur Dolmans, Louis Gehringer, Ina Klaric, Jia-Huei Ju, and Mohammad Aliannejadi. "The Powerless Noise: How Experimental Settings Shape the Reported Power of Noise." In *Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval*, 2026. arxiv.org/...2607.03615
61. ^Reddy, Pavan, and Aditya Sanjay Gujral. "EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System." AAAI Fall Symposium Series, 2025. arxiv.org/...2509.10540
62. ^RoyChowdhury, Ayush, Mulong Luo, Prateek Sahu, Sarbartha Banerjee, and Mohit Tiwari. "ConfusedPilot: Confused Deputy Risks in RAG-based LLMs." arXiv, August 2024. arxiv.org/...2408.04870
63. ^Morris, John X., Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M. Rush. "Text Embeddings Reveal (Almost) As Much As Text." In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 2023. arxiv.org/...2310.06816