CtrlK
Retrieval-augmented generation (RAG) is a technique in natural language processing that combines an information retrieval system with a generative model to produce text grounded in externally sourced knowledge. Rather than relying solely on the parameters of a large language model (LLM), a RAG system first retrieves relevant documents from a knowledge base and then feeds those documents to the model as additional context for generating a response. This approach addresses several well-known limitations of standalone LLMs, including hallucination, outdated training data, and the inability to cite sources.
The term was introduced in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research (now Meta AI), and the technique has since become one of the most widely adopted methods for building knowledge-intensive AI applications. As of 2026, over 60% of enterprises integrating generative AI use some form of retrieval-augmented architecture [1].
The foundational paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," was published in May 2020 on arXiv and presented at NeurIPS 2020 in December of that year [2]. The authors were Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela, all affiliated with Facebook AI Research and University College London.
The paper proposed combining two components: a pre-trained sequence-to-sequence model (BART) as the parametric memory, and a dense vector index of Wikipedia as the non-parametric memory accessed through a neural retriever based on Dense Passage Retrieval (DPR) [2]. The authors explored two formulations. RAG-Sequence conditioned on the same retrieved passages across the entire generated sequence, while RAG-Token could use different passages for each generated token.
The results were striking. RAG models set new state-of-the-art results on three open-domain question answering benchmarks: Natural Questions, TriviaQA, and WebQuestions. The models also generated more specific, diverse, and factual language than parametric-only baselines on tasks like Jeopardy question generation and abstractive summarization [2].
After the original paper, the concept of retrieval-augmented generation evolved rapidly. In 2021 and 2022, researchers experimented with applying the idea beyond question answering, using it for dialogue systems, code generation, and multi-hop reasoning. The release of ChatGPT in late 2022 and the subsequent explosion of interest in LLMs accelerated adoption of RAG as a practical way to ground model outputs in real data.
By 2023, what is now called "naive RAG" became the standard approach: chunk documents, embed them, store the embeddings in a vector database, retrieve relevant chunks at query time, and pass them to the LLM. During 2024, the field moved toward "advanced RAG" with techniques like query rewriting, hybrid search, reranking, and iterative retrieval. By 2025 and into 2026, the paradigm shifted further toward modular and agentic RAG systems, where AI agents orchestrate multiple retrieval and generation steps in feedback loops [3].
A RAG system operates in three main phases: retrieval, augmentation, and generation.
When a user submits a query, the system converts it into a vector representation using an embedding model. This query vector is then compared against a pre-built index of document vectors stored in a vector database. The system retrieves the top-K documents (or document chunks) that are most semantically similar to the query.
The retrieval step may use dense retrieval (vector similarity search), sparse retrieval (keyword matching algorithms like BM25), or a hybrid approach that combines both. Modern production systems frequently employ hybrid search because dense retrieval captures semantic meaning while sparse retrieval excels at matching exact terms, identifiers, and domain-specific jargon [4].
The retrieved documents are inserted into the prompt alongside the user's original query. This is typically done by placing the retrieved context before or around the query in a structured prompt template. The augmentation step may also include metadata such as source titles, dates, or relevance scores to help the model assess the information.
In advanced RAG systems, this step can include reranking the retrieved documents using a cross-encoder model, filtering out irrelevant results, or compressing the context to fit within the model's context window.
The augmented prompt is sent to the LLM, which generates a response conditioned on both the query and the retrieved context. Because the model has access to specific, relevant documents, it can produce answers that are more accurate and verifiable than those generated from parametric knowledge alone. The system can also be configured to include inline citations pointing back to the source documents.
A production RAG system involves several interconnected components. The following sections describe each in detail.
Before any retrieval can happen, source documents must be processed and indexed. This involves parsing raw documents (PDFs, web pages, databases, internal wikis), cleaning the text, and splitting it into smaller segments called "chunks."
Chunking strategy has a significant impact on retrieval quality. Research from 2025 found that chunking quality constrains retrieval accuracy more than the choice of embedding model [5]. The main approaches include:
| Chunking strategy | Description | Pros | Cons |
|---|---|---|---|
| Fixed-size | Split text into chunks of a set token or character length (e.g., 512 tokens) with optional overlap | Simple to implement; predictable chunk sizes | Often splits sentences mid-thought; separates tables from labels |
| Sentence-based | Split at sentence boundaries | Preserves grammatical units | Sentences vary in informativeness; some are too short to be useful |
| Semantic | Use an embedding model to detect topic shifts and split accordingly | Preserves topical coherence; up to 9% recall improvement over fixed-size | More computationally expensive; requires tuning |
| Recursive / hierarchical | Split by document structure (headings, sections, paragraphs), falling back to smaller units | Respects document structure | Requires structured documents; complex to implement |
| Agentic / adaptive | An AI agent dynamically determines chunk boundaries based on content | Highest accuracy (87% vs 13% for fixed-size in one clinical study) | Slowest; most expensive; newest approach |
A common best practice is to use overlapping chunks (e.g., 20% overlap between adjacent chunks) to ensure that information at chunk boundaries is not lost.
Embedding models convert text chunks (and user queries) into dense vector representations that capture semantic meaning. The quality of embeddings directly affects retrieval accuracy.
As of early 2026, commonly used embedding models include OpenAI's text-embedding-3-large, Cohere's embed-v4, Google's Gecko, and Voyage AI's voyage-3-large. Benchmarks from 2025 showed Voyage-3-large outperforming OpenAI and Cohere embeddings by 9 to 20% on retrieval tasks, with Voyage AI supporting 32K-token context windows compared to 8K for OpenAI [6]. Open-source alternatives like BGE, E5, and GTE from the MTEB (Massive Text Embedding Benchmark) leaderboard also perform competitively.
The embedding dimension (typically 768 to 3072 for modern models) and the maximum input length of the model are practical considerations that affect both performance and storage costs.
Vector databases are specialized storage systems optimized for indexing and querying high-dimensional vectors. They are a core infrastructure component of RAG systems.
The most widely used vector databases as of 2026 include:
| Database | Type | Key strengths | Scale |
|---|---|---|---|
| Pinecone | Managed cloud | Ease of use; sub-50ms p99 latency; serverless option | Billions of vectors |
| Weaviate | Open-source / cloud | Hybrid search (vector + keyword); strong ecosystem | Billions of vectors |
| Milvus / Zilliz Cloud | Open-source / managed | Cost-efficient at scale; lowest latency in benchmarks | Billions of vectors |
| Qdrant | Open-source / cloud | Rust-based performance; sub-50ms p99 latency; filtering | Billions of vectors |
| Chroma | Open-source | Developer-friendly; lightweight; great for prototyping | Millions of vectors |
| pgvector | PostgreSQL extension | Uses existing Postgres infrastructure; no new system needed | Tens of millions of vectors |
| FAISS | Library (Meta) | Highly optimized for research; GPU support | Billions of vectors (in-memory) |
These databases typically use the HNSW (Hierarchical Navigable Small World) algorithm for approximate nearest neighbor search. HNSW builds a multi-layered graph structure that enables logarithmic search complexity, making it practical to query billions of vectors with latencies in the 10 to 100 millisecond range [7].
The retrieval component of a RAG system can employ several different algorithms, often in combination.
Sparse retrieval (BM25): BM25 (Best Match 25) is a classical information retrieval algorithm based on term frequency and inverse document frequency. It excels at exact keyword matching and delivers millisecond response times at millions of documents without requiring GPU infrastructure [4].
Dense retrieval: Dense retrieval uses neural embedding models to represent both queries and documents as vectors in a continuous space. Similarity is computed using cosine similarity or dot product. This approach captures semantic relationships that keyword matching misses, but it can struggle with exact identifiers, acronyms, and rare terms.
Hybrid search: Hybrid search runs sparse and dense retrievers in parallel on the same query, then merges results using a fusion algorithm. The most common fusion method is Reciprocal Rank Fusion (RRF), which combines rankings from each retriever by summing the reciprocal of each document's rank. Hybrid search consistently outperforms either method used alone [4].
Reranking: After initial retrieval, a reranker (typically a cross-encoder model) scores each query-document pair jointly. Cross-encoders are more accurate than embedding similarity because they process the query and document together, but they are too slow for first-stage retrieval over large collections. Models like cross-encoder variants trained on MS MARCO consistently improve retrieval metrics like NDCG and MRR [8].
As the field has matured, several distinct RAG architectures have emerged, each addressing different limitations.
Naive RAG is the simplest implementation: documents are chunked, embedded, and stored in a vector database. At query time, the user's question is embedded, the top-K similar chunks are retrieved, and they are passed directly to the LLM for answer generation. There is no query transformation, no reranking, and no verification of the generated answer.
This approach was the standard in 2023, but its limitations quickly became apparent in production settings. Problems include irrelevant retrieval results, lost context from poor chunking, and no mechanism to handle ambiguous or multi-faceted queries. By 2025, naive RAG was widely considered inadequate for production-grade applications [3].
Advanced RAG introduces multiple optimization techniques at each stage of the pipeline:
Advanced RAG became the dominant production approach during 2024 and remains widely used [9].
Modular RAG treats the system as a collection of interchangeable components (retrievers, generators, evaluators, routers) that can be composed and configured for different tasks. Rather than a fixed linear pipeline, modular RAG allows components to be swapped, added, or removed based on the specific use case.
For example, a modular system might route simple factual queries through a lightweight retrieval path while sending complex analytical queries through a multi-step retrieval and reasoning pipeline. This flexibility is particularly valuable in enterprise settings where different departments have different requirements [3].
GraphRAG, introduced by Microsoft Research in 2024, combines vector search with knowledge graphs to capture relationships between entities. Instead of (or in addition to) retrieving flat text chunks, GraphRAG constructs a graph of entities and their relationships from the source documents. During retrieval, the system traverses the graph to find connections that vector similarity search alone would miss.
This approach is particularly effective for queries that require understanding relationships, hierarchies, or multi-hop reasoning. For example, answering "Which subsidiaries of Company X operate in the healthcare sector?" requires traversing organizational relationships that flat chunk retrieval handles poorly. Early benchmarks suggest GraphRAG implementations can achieve search precision up to 99% for complex, multi-layered corporate queries [10].
Agentic RAG represents the newest evolution, combining RAG with autonomous agents. Instead of following a fixed retrieval pipeline, an AI agent plans its approach, decides what information it needs, executes multiple retrieval steps (potentially across different data sources and tools), evaluates intermediate results, and iterates until it has sufficient information to generate a high-quality answer.
In an agentic RAG system, the agent might:
This approach mirrors how a skilled human researcher works, and it can handle complex, open-ended questions that fixed pipelines cannot. Agentic RAG became the cutting-edge approach in 2025 and is increasingly adopted in enterprise settings [3].
Self-RAG (Self-Reflective Retrieval-Augmented Generation), proposed by Asai et al. in 2023, adds a self-reflection mechanism to the generation process. The model dynamically decides whether retrieval is needed for a given query, evaluates the relevance of retrieved passages, and critiques its own generated output. Special reflection tokens are trained into the model to enable these decisions. This allows the system to avoid unnecessary retrieval for simple queries while ensuring thorough retrieval for knowledge-intensive ones [11].
Corrective RAG introduces a verification step that evaluates retrieved documents before they are used for generation. If the retriever returns low-confidence results, the system can trigger a web search or alternative retrieval strategy rather than generating an answer from potentially irrelevant context. This acts as a safety net against poor retrieval quality [11].
RAG and fine-tuning are two distinct approaches to customizing LLM behavior, and they serve different purposes.
| Dimension | RAG | Fine-tuning |
|---|---|---|
| What it changes | The information available to the model at inference time | The model's internal weights and behavior |
| Best for | Injecting up-to-date or proprietary knowledge; factual Q&A over specific documents | Changing output style, format, or tone; domain-specific reasoning patterns |
| Data freshness | Can access real-time or frequently updated data | Knowledge is frozen at training time |
| Cost | Lower upfront cost; requires vector database infrastructure | Higher upfront cost for training; lower per-query inference cost |
| Latency | Additional retrieval step adds latency (typically 100-500ms) | No retrieval overhead; sub-second responses |
| Hallucination risk | Reduced (grounded in retrieved documents) | Can still hallucinate if training data is insufficient |
| Transparency | Can cite specific source documents | Difficult to trace where information came from |
| Maintenance | Update knowledge by updating the document index | Requires retraining to update knowledge |
The two approaches are not mutually exclusive. A growing trend as of 2025 is RAFT (Retrieval-Augmented Fine-Tuning), which combines both techniques. An organization might fine-tune a model to become an expert in medical terminology and diagnostic reasoning, then deploy it in a RAG architecture that provides access to the latest medical research papers and patient records [12]. This hybrid approach leverages fine-tuning for domain expertise and RAG for access to current, specific information.
One of the primary motivations for RAG is reducing the tendency of LLMs to generate plausible but incorrect information. By grounding generation in retrieved documents, RAG constrains the model's output to information that actually exists in the knowledge base. Research in healthcare contexts found that chatbots using RAG with reliable reference sources showed hallucination rates of 0% for GPT-4 and 6% for GPT-3.5, compared to approximately 40% for conventional chatbots without RAG [13]. The MEGA-RAG framework achieved a reduction in hallucination rates by over 40% compared to baseline models [14].
LLMs have a knowledge cutoff determined by their training data. RAG overcomes this limitation by connecting the model to continuously updated document stores. When new information is added to the knowledge base, it becomes immediately available for retrieval without any model retraining. This is particularly valuable in domains like news, finance, legal, and medicine where information changes frequently.
RAG systems can provide citations linking each part of a generated answer to its source documents. This transparency allows users to verify claims, assess the reliability of sources, and build trust in the system's outputs. Source attribution is a requirement in many enterprise and regulated environments where accountability matters.
RAG enables a general-purpose LLM to become an expert in any domain simply by connecting it to the relevant knowledge base. A single model can serve as a legal research assistant, a medical information system, or a technical support agent depending on which document collection it retrieves from. This avoids the need to train or fine-tune separate models for each domain.
Compared to fine-tuning or training custom models, RAG is generally more cost-effective. Organizations can use off-the-shelf LLMs and invest in document indexing infrastructure instead. Updating knowledge requires only re-indexing documents rather than expensive retraining runs.
Poor chunking is one of the most common causes of RAG failure. Fixed-size chunking can split tables from their headers, separate code from its comments, or break a coherent argument across two chunks. The traditional approach of using a single granularity for both embedding and retrieval creates a structural conflict: small chunks produce precise embeddings but may lack context, while large chunks preserve context but dilute the embedding signal [5]. Finding the right chunking strategy for a given corpus often requires significant experimentation.
Even with good embeddings, retrieval can fail in several ways. Semantic similarity in embedding space does not always correspond to actual relevance, particularly for domain-specific terminology. The retriever may return passages that are topically related but do not answer the specific question. Missing or unprocessed documents in the index create blind spots. Multi-hop questions that require synthesizing information from several documents are particularly challenging for single-step retrieval.
Although context windows have grown dramatically (from 4K tokens in early GPT-3.5 to 1 million or more tokens in Gemini models), there are practical limits to how much retrieved context a model can effectively use. Research from Chroma in July 2025 tested 18 models including GPT-4.1, Claude 4, and Gemini 2.5 and found that retrieval performance degrades as context length increases [15]. Counterintuitively, shorter and more precise context often produces better answers than inserting 50,000 tokens of retrieved text. Most production systems target assembled context under 8,000 tokens per query.
The retrieval step adds latency to every request. A typical RAG pipeline involves embedding the query (10-50ms), searching the vector database (10-100ms), optionally reranking results (50-200ms), and then generating the response. Production RAG applications generally target under 2 seconds for end-to-end response time [5]. Reranking, while improving precision, is the most expensive step and may need to be omitted for latency-sensitive applications.
In enterprise settings, RAG systems must enforce access controls to ensure users only retrieve documents they are authorized to see. A customer support agent should not retrieve internal HR documents, and a junior employee should not access board-level financial data. Implementing document-level permissions in vector databases adds complexity to the architecture.
Measuring the quality of a RAG system is more complex than evaluating a standalone model. Teams must assess retrieval quality (are the right documents being found?), generation quality (is the answer correct and well-formed?), and faithfulness (does the answer accurately reflect the retrieved documents rather than hallucinating?). Frameworks like RAGAS (Retrieval Augmented Generation Assessment) have emerged to address this challenge, but evaluation remains an active area of research.
RAG has found adoption across numerous industries and use cases.
Enterprise search is one of the most common RAG applications. Companies use RAG to build internal assistants that can answer employee questions by searching across internal wikis, documentation, Slack messages, email archives, and databases. Glean, a prominent enterprise search company, uses RAG as a core part of its product. Businesses using RAG-powered enterprise search report up to a 40% increase in content accuracy and a 50% reduction in research time [1].
RAG-powered chatbots and agent assistants can retrieve relevant knowledge articles, past cases, and technical documentation to resolve customer issues faster. DoorDash uses a RAG-based chatbot that condenses customer conversations, searches its knowledge base for relevant articles and resolved cases, and generates contextually appropriate responses [16]. LinkedIn's RAG implementation achieved a 28.6% reduction in median customer support resolution time [16].
Law firms use RAG to query vast legal databases, ensuring that AI-generated responses are grounded in current regulations and case law. Applications include clause comparison across contracts, obligations extraction, and audit trail generation with source links for human review. A Stanford study found that legal RAG can reduce hallucinations compared to general-purpose AI systems, though hallucinations remain a significant concern in legal contexts where accuracy is paramount [17].
Healthcare providers use RAG systems that draw from electronic health records, clinical guidelines, and medical literature to support clinical decision-making. RAG helps ensure that medical AI applications provide recommendations grounded in current evidence rather than potentially outdated training data. Healthcare sector adoption of multimodal RAG for medical policy compliance and regulatory reference systems has produced 25 to 30% improvements in compliance accuracy and processing efficiency [18].
Financial institutions use RAG for regulatory compliance, risk assessment, and investment research. RAG systems can retrieve and synthesize information from earnings calls, SEC filings, market reports, and internal analysis to support decision-making with up-to-date, cited information.
Developers use RAG-powered coding assistants that retrieve relevant code snippets, documentation, and API references from a codebase. This helps generate more accurate code suggestions that are consistent with existing patterns and conventions in a project.
Several open-source frameworks have emerged to simplify RAG development.
LangChain is the most widely adopted framework for building LLM-powered applications, including RAG systems. Originally released in October 2022 by Harrison Chase, it provides abstractions for document loading, text splitting, embedding, vector storage, retrieval, and chain composition. The LangChain ecosystem saw 220% growth in GitHub stars and a 300% increase in downloads between Q1 2024 and Q1 2025 [19]. LangGraph, an extension for building stateful, multi-agent workflows, reached version 1.0 in October 2025 and has become the recommended approach for complex RAG pipelines within the LangChain ecosystem.
LlamaIndex (originally GPT Index) focuses specifically on connecting LLMs with external data. It provides specialized abstractions for hierarchical chunking, auto-merging retrieval, sub-question decomposition, and built-in reranking. In 2025, LlamaIndex achieved a 35% boost in retrieval accuracy and added a Workflows system for complex multi-step agents [19]. Benchmarks show LlamaIndex achieves document retrieval speeds 40% faster than LangChain for comparable tasks.
Haystack, developed by deepset, is a modular framework with a strong focus on production readiness. It provides components for document retrieval, question answering, and text summarization, and supports a wide range of document stores including Elasticsearch and FAISS. Haystack is known for its testable pipeline architecture with clear component contracts. In production benchmarks, it achieved 99.9% uptime and showed the lowest framework overhead (~5.9ms) and token usage (~1.57K) among the major frameworks [19].
Additional notable tools in the RAG ecosystem include:
The rapid expansion of LLM context windows has raised questions about whether RAG will remain necessary. Gemini 1.5 Pro introduced a 1-million-token context window in early 2024, and subsequent models have pushed this further. One million tokens is roughly equivalent to 750,000 words, or about 3,000 pages of text. In principle, this allows users to feed entire document collections directly into the model without a retrieval step.
However, practical experience tells a more nuanced story. A Gartner Q4 2025 survey of 800 enterprise AI deployments found that 71% of companies that initially deployed "context-stuffing" approaches (loading entire document sets into the context window) added vector retrieval layers within 12 months [15]. Several factors drive this:
The emerging consensus as of 2026 is that long-context windows complement rather than replace RAG. Long context windows are useful for RAG systems themselves, allowing them to hold more complete, semantically coherent retrieved chunks or to aggregate intermediate results for multi-step retrieval. This marks a shift toward what practitioners are calling "Context Engineering," which combines intelligent retrieval with the expanded context capacities of modern models [20].
For smaller datasets that fit within a context window (a single codebase, a specific document collection under a few thousand pages), direct context loading can be practical. For enterprise-scale applications with millions of documents, continuously updated data, or strict latency and cost requirements, RAG remains the standard approach.
The global RAG market was valued at approximately $1.2 billion in 2024 and is forecast to reach $11 billion by 2030, representing a compound annual growth rate of 49.1% [1]. Major cloud providers including AWS, Google Cloud, and Microsoft Azure all offer managed RAG services. The technology has moved from experimental to a core component of enterprise AI infrastructure.