# Information Retrieval

> Source: https://aiwiki.ai/wiki/information_retrieval
> Updated: 2026-08-01
> Fact-checked: 2026-08-01
> Categories: Information Retrieval, Machine Learning, Natural Language Processing
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Information Retrieval." aiwiki.ai, 1 Aug 2026. https://aiwiki.ai/wiki/information_retrieval
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Information retrieval** (IR) is the study and engineering of systems that identify material likely to satisfy an information need. The material may be a document, passage, image, record, or other retrievable unit. A user often expresses the need as a short query, but the query is only an imperfect description of what would be useful. An IR system therefore has to represent the collection and query, generate candidates efficiently, estimate relevance, rank results, and evaluate whether the ranking serves the intended task. The standard textbook distinction is between retrieving items from a largely unstructured collection and answering a formally specified database query whose correct result is defined by a schema and query language.[1]

IR supplies core methods for a [search engine](https://aiwiki.ai/wiki/search_engine), document discovery, enterprise search, [question answering](https://aiwiki.ai/wiki/question_answering), and [retrieval-augmented generation](https://aiwiki.ai/wiki/retrieval_augmented_generation). It overlaps with [natural language processing](https://aiwiki.ai/wiki/natural_language_processing) and [machine learning](https://aiwiki.ai/wiki/machine_learning), but it is not reducible to either. An encoder or language model is only one component of a retrieval system. Collection design, indexing, candidate generation, relevance assessment, ranking objectives, latency constraints, and presentation all affect the result.[1]

There is no universally best retriever. Effectiveness depends on the population of information needs, corpus, judgment policy, metric, cutoff depth, training data, and resource budget. Exact identifiers may favor lexical matching; paraphrased questions may favor learned representations; and a method that leads one benchmark may fail under domain or language shift. For that reason, defensible claims compare named systems under a specified protocol rather than calling an architecture inherently more accurate.[1]

## Historical foundations

The vector-space model represents documents and queries as weighted term vectors and ranks documents by a similarity function. Gerard Salton, Anita Wong, and Chung-Shu Yang described this framework in a 1975 paper using SMART experiments. Their formulation separated document representation from a comparison function, making it possible to study alternative term weights and similarity measures within one retrieval framework.[2]

Karen Sparck Jones's 1972 study gave a statistical account of term specificity. Terms occurring in few documents were treated as more discriminative than terms distributed throughout the collection. This document-frequency signal became the inverse-document-frequency component used in many term-weighting schemes. The paper tested variants experimentally; it did not define one immutable formula that every later system must use.[3]

Probabilistic retrieval models instead motivate ranking through evidence about relevance. Robertson and Zaragoza's review traces the binary independence model, relevance weighting, and the Okapi line that produced BM25. BM25 combines a query-term weight with saturating within-document term frequency and document-length normalization. Its parameters and inverse-document-frequency variant are implementation and collection choices, so values such as `k1 = 1.2` and `b = 0.75` are common starting points, not universal defaults or guarantees.[4]

Web retrieval added signals beyond document text. Brin and Page's account of the early Google system described crawling, anchor text, an inverted index, and PageRank, which used the Web's link graph as one ranking signal. Modern web ranking systems are not characterized by PageRank alone, and this historical paper does not establish how any current commercial engine weights its signals.[5]

These strands remain visible in contemporary systems: statistical term weighting, probabilistic ranking, link or graph evidence, supervised features, and learned text representations can all contribute scores. Their coexistence is important. The history of IR is not a simple replacement sequence in which dense retrieval made lexical indexes obsolete.[1][4]

## Retrieval task and relevance

An IR task starts with a **collection**, a definition of the retrievable unit, and a population of information needs. A unit might be a whole document, a passage, a product, or a previously segmented chunk. Changing the unit changes both the index and the meaning of relevance. A passage can directly answer a question even when its parent document contains substantial unrelated material; a document-level legal search may instead require the full record.[1]

The user's query is evidence about an information need, not a complete specification of it. Queries can be ambiguous, underspecified, misspelled, multilingual, or dependent on time and context. Systems may normalize text, correct spelling, detect phrases, expand terms, apply filters, or use interaction history. Each intervention can help one query class and harm another, so query processing should be evaluated as part of the end-to-end system.[1]

**Relevance** is operationalized through a judgment policy. A study might ask assessors whether a result is topically related, whether it contains an answer, or whether it would be useful for completing a task. Judgments can be binary or graded. They can also vary between assessors and with the information supplied to an assessor. A reported metric is therefore relative to the collection, topic statements, judgment scale, and assessor process that produced it.[1]

Retrieval is commonly modeled as ranking. For a query `q` and candidate `d`, a scoring function assigns `s(q,d)`, and results are ordered by that value. Scores from different models need not be probabilities or comparable across systems. A BM25 score, vector dot product, and cross-encoder logit can have unrelated scales even when larger values all indicate a stronger match.[4]

Some applications require set retrieval rather than a short ranked list. Systematic review, patent search, legal discovery, and safety investigations may emphasize high recall and auditable query logic. Navigational search and factoid question answering may emphasize the first useful result. These objectives justify different metrics and operating points; optimizing one should not be presented as optimizing all retrieval use cases.[1]

## Collection preparation and indexing

Collection preparation determines what a system is capable of returning. Ingestion may fetch files, parse formats, remove boilerplate, identify language, extract metadata, and split content into retrievable units. A parser that silently omits tables or scanned text creates a recall failure before ranking begins. Corpus snapshots and parser versions are therefore part of an experiment, not incidental infrastructure.[1]

Stable document identifiers are necessary for judgments, updates, and deletion. Deduplication must distinguish byte-identical copies, near duplicates, and legitimately repeated passages. If training and test collections share duplicates, measured effectiveness can be inflated. If a production index keeps obsolete copies, a user may see stale or conflicting results even when the ranker behaves as designed.[1]

An inverted index stores a vocabulary and postings that identify documents containing each term. Positions support phrase and proximity queries; field identifiers permit different treatment of titles and bodies; and stored corpus statistics support scoring. Dense retrieval instead stores learned vectors and an index over a chosen distance or similarity measure. Many systems maintain both representations because they encode different evidence.[1]

Text analysis is language and domain dependent. Token boundaries, Unicode normalization, case, diacritics, compound words, stemming, and stop words can all change matching. Character or subword representations reduce some vocabulary problems without eliminating them. The correct analyzer is the one validated for the intended documents and queries, including exact strings that must not be normalized away.[1]

Indexing also defines update semantics. An application should specify when a new or changed source becomes searchable, how deletions propagate, whether document statistics update immediately, and whether old embeddings remain after an encoder change. Without versioned corpus and index identities, a result cannot be reproduced even if the ranking code is unchanged.[1]

## Test collections and experimental evaluation

The Text REtrieval Conference (TREC) began in 1992 as part of the TIPSTER program, with the U.S. National Institute of Standards and Technology providing infrastructure for shared retrieval experiments. TREC established a recurring model in which organizers define tasks and collections, participants submit runs, and results are evaluated against common judgments. Its tracks have changed over time, so "TREC performance" is not a single enduring task or metric.[6]

For many TREC collections, relevance judgments are distributed as **qrels**, which associate a topic and document identifier with an assessor label. NIST documents that evaluation uses these judgments to score submitted result files. A qrels file is not a declaration that every unjudged document is nonrelevant; it records the documents that were assessed under that collection's procedure.[7]

Judging every document for every topic is usually infeasible. Pooling selects documents from participating systems for assessment, leaving the rest unjudged. Buckley and colleagues showed that pools that are too small relative to a large collection can favor documents containing topic-title words. The finding is a warning about collection construction, not proof that every pooled test collection is unusable.[8]

Binary judgments do not express degrees of usefulness. Jarvelin and Kekalainen developed cumulative-gain measures for graded relevance and rank discounting. Normalized discounted cumulative gain (NDCG) compares a system's discounted gain with the ideal ordering for the same judged results. Its value depends on the gain mapping, discount function, cutoff, and treatment of unjudged items, which should accompany any reported score.[9]

Incomplete judgments affect metrics differently. Buckley and Voorhees found that conventional measures could become unreliable when judgments were substantially incomplete and introduced bpref, which compares judged relevant documents with judged nonrelevant documents. Bpref is not a universal substitute for average precision; it addresses a particular evaluation problem and still depends on the available judgments.[10]

The 2019 TREC Deep Learning Track illustrates why protocol details matter. It used MS MARCO-derived document and passage tasks, large training sets, blind TREC-style test evaluation, and 43 judged test queries for each task. Neural language-model runs significantly outperformed traditional runs in that edition, but the organizers explicitly noted possible explanations including abundant training data and the presence of neural runs in the judging pools. The result should not be generalized to every collection or query distribution.[11]

### Common metrics

| Metric | What it summarizes | Important boundary |
| --- | --- | --- |
| [Precision](https://aiwiki.ai/wiki/precision) at `k` | Fraction of the first `k` retrieved items judged relevant | Does not measure missed relevant items beyond the cutoff |
| [Recall](https://aiwiki.ai/wiki/recall) at `k` | Fraction of judged relevant items found in the first `k` | Requires a usable denominator and is affected by incomplete judgments |
| Average precision and MAP | Precision at ranks containing relevant items, averaged within and then across topics | Usually assumes binary relevance and a stated treatment of unjudged items |
| Reciprocal rank and MRR | Reciprocal rank of the first relevant item, averaged across topics | Ignores relevant items after the first |
| NDCG at `k` | Discounted graded gain normalized by an ideal ranking | Depends on grades, gain, discount, cutoff, and judgment completeness |

Metrics should be computed per query before aggregation unless the definition specifies otherwise. Means can hide regressions on rare query classes or languages. Confidence intervals, paired significance tests, and per-slice results help distinguish a stable improvement from noise. Repeated tuning on a public leaderboard can also overfit the test distribution even when the model never trains directly on its labels.[1][9]

Offline relevance does not capture every user outcome. Latency, result diversity, freshness, comprehension, and task completion may matter. Behavioral measures require an experimental design and should be interpreted alongside offline judgments rather than treated as interchangeable with them.[1]

## Learning to rank and interaction data

**Learning to rank** estimates a ranking function from labeled or behavioral data. Pointwise methods predict a label or score for each query-document pair. Pairwise methods learn preferences between two documents. Listwise methods optimize an objective defined over a ranked list. The distinctions concern training objectives; a deployed system may still score candidates individually or in batches.[12]

RankNet is an influential pairwise method. Burges and colleagues modeled the probability that one item should rank above another and minimized cross-entropy on pairwise preferences. Its experiments covered ranking datasets available to the authors; the paper does not imply that pairwise loss is optimal for every IR metric or modern neural architecture.[12]

Behavioral logs provide scale but introduce selection effects. A result must be shown and examined before it can be clicked, and the existing ranker determines much of that exposure. Joachims and coauthors showed experimentally that clickthrough observations can support relative preference inferences under specified examination assumptions. The result does not make every click an unbiased relevance label.[13]

Later work developed an inverse-propensity approach that learns from biased click observations under explicit assumptions about examination. Such correction requires logging or estimating propensities and does not remove every source of confounding. A model trained on clicks generated by an earlier system can reproduce that system's exposure bias while appearing accurate on similarly collected data.[14]

Supervised rankers can combine lexical scores, field matches, freshness, authority, neural scores, and context. Feature availability at serving time, leakage across train and test periods, and feedback loops must be audited. Training and serving pipelines must compute each feature with the same definition, or an apparent offline gain can disappear after deployment.[12][14]

Rank fusion offers an unsupervised alternative when systems return separate lists. Reciprocal rank fusion (RRF) assigns each document the sum of `1 / (k + rank)` over input rankings. Cormack, Clarke, and Buettcher evaluated the method on specific TREC and LETOR experiments. The commonly reused `k = 60` came from their pilot work; it is not a theoretical constant, and RRF is not guaranteed to improve arbitrary inputs.[15]

Score fusion and rank fusion answer different calibration problems. A weighted score sum can use magnitude information but requires comparable or normalized scores. RRF discards magnitude and uses positions, which makes heterogeneous lists easier to combine but can reward agreement between correlated systems. Fusion depth, duplicate handling, missing-document treatment, and weights should be recorded for reproducibility.[15]

## Lexical and probabilistic retrieval

An **inverted index** maps terms to postings lists of documents, often with term frequency, positions, fields, or other payloads. Index construction usually includes parsing, tokenization, and optional normalization such as case folding, stemming, or stop-word handling. These choices define what can match. For identifiers, source code, chemical names, or morphologically rich languages, an analyzer designed for general English prose can destroy useful distinctions.[1]

A Boolean system combines posting lists with operations such as AND, OR, and NOT. Ranked lexical retrieval instead accumulates term contributions. In a vector-space implementation, document and query term weights can be compared with cosine similarity or another function. There is no single canonical "TF-IDF algorithm": choices include raw or transformed term frequency, smoothed inverse document frequency, normalization, and field weighting.[1][2][3]

BM25 gives repeated occurrences diminishing returns and adjusts term contribution according to document length relative to the collection average. The standard review presents a family of related models, including variants for multiple fields. A score is only rank-equivalent within a fixed query and implementation; treating it as a calibrated probability of relevance is generally incorrect.[4]

Lexical methods are especially useful when exact strings carry meaning, including names, model numbers, error codes, citations, and quoted phrases. Their limitations include vocabulary mismatch and weak handling of paraphrase. Query expansion and relevance feedback can reduce mismatch, but expansion terms can also shift the query away from the original need.[1][4]

Efficiency comes from traversing only relevant postings and from skipping documents that cannot enter the current top results. Compression reduces memory and I/O, while block metadata can support dynamic pruning. Index layout, compression, caching, and query distribution influence latency as much as the abstract scoring formula does.[1]

## Neural retrieval architectures

Neural IR uses learned representations or interaction functions to estimate query-document relevance. "Neural retrieval" covers architectures with very different serving costs. A method may learn a representation, an interaction function, term weights, or several of these together, so architecture and serving protocol must be named.[1]

A **dual encoder** encodes the query and document separately. Document vectors can be computed before the query arrives, and a vector index can retrieve candidates by similarity. Sentence-BERT demonstrated this efficiency distinction for semantic similarity: independent embeddings support comparison without running a transformer jointly for every pair. A compact vector is useful for search, but it is an information bottleneck and its behavior depends on training examples and negatives.[16]

A **cross-encoder** processes a query and candidate together, allowing attention across both sequences. Nogueira and Cho applied BERT to rerank passages returned by a first-stage system and reported gains on their TREC CAR and MS MARCO experiments. Because the document representation depends on the query, cross-encoder scores cannot ordinarily be precomputed for a whole corpus. Cross-encoders are therefore commonly used on a bounded candidate set, not assumed to be feasible for exhaustive first-stage search.[17]

Dense Passage Retrieval (DPR) trained separate question and passage encoders with contrastive examples for open-domain question answering. On the paper's tested QA datasets, DPR improved top-20 passage retrieval accuracy over its Lucene-BM25 baseline by 9 to 19 absolute percentage points. That result is bounded to those datasets, corpus construction, training process, and retrieval-accuracy metric; it is not evidence that dense retrieval always beats BM25.[18]

**Late interaction** retains more token-level information while preserving offline document encoding. ColBERT encodes query and document tokens separately and sums, over query tokens, the maximum similarity to any document token. Its paper reported competitive effectiveness and lower query-time computation than the compared BERT rerankers under its experimental setup. Storage and indexing costs are higher than for a single vector because multiple vectors represent each document.[19]

ColBERTv2 introduced residual compression and denoised supervision. Its authors reported a six-to-ten-fold reduction in the space used for late-interaction representations and evaluated both in-domain and out-of-domain retrieval. These are results for the paper's implementation and compression settings, not a fixed storage ratio for every tokenizer, collection, or index.[20]

**Learned sparse retrieval** predicts weights in a vocabulary-sized space that can be served through an inverted index. SPLADE uses a masked-language-model head, pooling, and sparsity regularization to learn both weighting and expansion. A predicted term may be absent from the source text, which can bridge vocabulary mismatch but makes the representation less directly attributable than ordinary term counts.[21]

### Architecture comparison

| Architecture | Offline document work | Query-time interaction | Typical role and constraint |
| --- | --- | --- | --- |
| Lexical sparse | Build postings from analyzed terms | Accumulate matching postings | Efficient first stage; exact vocabulary can miss paraphrase |
| Learned sparse | Predict sparse term weights, then index them | Accumulate predicted-term postings | First stage with expansion; training and larger postings may add cost |
| Single-vector dual encoder | Encode one vector per unit | Nearest-neighbor similarity | Efficient semantic candidates; fixed-vector bottleneck |
| Late interaction | Store multiple token vectors per unit | Token-level maximum similarities | Richer matching; more storage and scoring work |
| Cross-encoder | Usually no reusable document score | Jointly encode each query-candidate pair | Strong reranking option; cost grows with candidates and sequence length |

No row has an invariant quality ordering. A first-stage retriever can only pass retrieved candidates to later stages, so reranking cannot recover a relevant item that candidate generation omitted. Conversely, increasing candidate depth can raise recall while also increasing latency and reranking cost. End-to-end evaluation should vary both candidate depth and final display cutoff.[17][19]

## Approximate nearest-neighbor search

Exhaustively comparing a query vector with every document vector is exact but costly for a large collection. Approximate nearest-neighbor (ANN) methods trade some recall for lower latency or memory. The relevant "recall" here is ANN recall relative to exact vector neighbors, which is different from relevance recall measured against human judgments.[1]

Hierarchical Navigable Small World (HNSW) constructs a multilayer proximity graph and searches from sparse upper layers toward a denser base layer. Malkov and Yashunin reported logarithmic scaling behavior in their experiments and exposed construction and search parameters that trade memory, build cost, query work, and neighbor recall. HNSW does not guarantee identical results to exhaustive search.[22]

Faiss is a library for dense-vector similarity search and clustering. Johnson, Douze, and Jegou described GPU implementations and indexing schemes including inverted files and product quantization. Faiss is a library rather than a database: persistence, filtering semantics, replication, access control, and application-level consistency must be provided by the surrounding system when required.[23]

ANN-Benchmarks compared algorithms across datasets using recall versus query time and included build time and index size in its methodology. Its central lesson is methodological: results depend on hardware, dataset geometry, distance function, parameter tuning, and the point selected on the recall-speed frontier. A vendor's latency number without those controls is not a transferable performance guarantee.[24]

Vector compression can reduce memory and improve cache behavior at the cost of distortion. Product quantization represents vectors through learned codebooks; lower-precision or binary representations make different tradeoffs. Filtering can also alter ANN behavior because applying metadata constraints before, during, or after graph traversal changes both work and reachable candidates.[23][24]

## Benchmarks and generalization

MS MARCO began as a machine-reading-comprehension dataset built from anonymized Bing queries and human-generated answers. The original paper reports 1,010,916 queries and 8,841,823 passages extracted from 3,563,535 web documents. Those figures describe that release; later passage-ranking and document-ranking tasks have their own files, splits, and label distributions.[25]

Official MS MARCO documentation separates question-answering datasets from ranking datasets and publishes task-specific downloads and evaluation instructions. This distinction prevents a common error: describing all one-million-plus queries as passage-ranking training labels. Sparse judgments in ranking data also mean that an unjudged passage may be useful even when evaluation treats it as nonrelevant.[26]

BEIR assembled 18 public datasets spanning nine retrieval tasks and compared ten retrieval systems under a zero-shot setup. The paper found BM25 to be a robust baseline, while late-interaction and reranking approaches achieved the best average zero-shot effectiveness among those tested at higher computational cost. Dataset-level results varied, so the average does not establish one winner for every domain.[27]

The Massive Text Embedding Benchmark (MTEB) broadened evaluation beyond retrieval. Its EACL paper covers eight embedding tasks, 58 datasets, and 112 languages. A model's aggregate MTEB rank mixes task types and dataset scales; for an IR deployment, retrieval subsets and target-domain results are more informative than the overall average.[28]

The multilingual extension MMTEB reports more than 500 tasks across more than 250 languages in its paper. Language coverage is highly uneven, and a dataset's inclusion does not establish equal quality or sufficient test power for every language. Claims of support should name the evaluated language, script, domain, and task rather than inherit the benchmark's total language count.[29]

Mr. TyDi evaluates monolingual retrieval in eleven typologically diverse languages. Its experiments found that its BM25 baseline outperformed the evaluated multilingual dense retriever. The sparse-dense hybrid had higher average MRR@100 and recall@100 than tuned BM25, with the MRR improvement statistically significant in nine of eleven languages under the paper's paired test. This result supports testing complementary signals on that benchmark; it does not prove that hybrid retrieval is universally best.[30]

MIRACL provides retrieval data across eighteen languages, with topics and relevance judgments developed through native speakers. The paper documents heterogeneous corpora, language-specific resources, and baselines. Cross-language averages can conceal large differences in corpus size, script, tokenization, and available training data, so per-language reporting is essential.[31]

Benchmark integrity requires recording the exact corpus and qrels version, query fields, text preprocessing, allowed training data, model checkpoint, candidate depth, ANN configuration, reranker depth, metric implementation, and random seeds. It also requires preventing overlap between evaluation queries or documents and training data when the protocol forbids it. A leaderboard score without this provenance is difficult to reproduce or interpret.[11][26][27]

## Multistage and hybrid retrieval

Large systems often use a cascade. A low-cost stage generates candidates, later stages compute more expensive features, and a final policy may diversify, filter, or group results. Cascades make latency controllable, but they distribute error: candidate generation determines the ceiling for downstream recall, and later stages determine which candidates remain visible.[17]

**Hybrid retrieval** combines signals such as BM25 and dense similarity. Parallel retrieval can improve coverage when the systems make complementary errors. It can also degrade results if one input is weak, correlated, or improperly weighted. Whether hybrid retrieval helps is an empirical question for the target query set, not a production default justified by architecture alone.[15][27][30]

Raw-score interpolation requires a calibration rule because lexical and vector scores have different ranges and distributions. Rank fusion such as RRF avoids direct scale comparison, but its outcome still depends on list depth and the fusion constant. A learned combiner needs representative labels and should be evaluated for drift and exposure bias.[14][15]

Reranking depth is a resource allocation decision. A cross-encoder may improve ordering among 100 candidates while increasing tail latency; reducing the depth may be faster but omit items that the reranker would have promoted. Batching, sequence truncation, caching, and hardware affect the actual tradeoff. Model parameter count alone is not an adequate latency estimate.[17]

Diversification may be needed when a query has multiple intents. A list containing near-duplicate passages can score well under some relevance metrics while providing little additional information. Diversity objectives should be stated explicitly because they can trade topical redundancy against coverage and may require subtopic judgments absent from a conventional qrels file.[1]

## Retrieval in question answering and RAG

Retrieval-Augmented Generation (RAG) combines a retriever with a sequence generator. Lewis and colleagues evaluated variants in which retrieved passages were latent evidence for knowledge-intensive NLP tasks and reported state-of-the-art results on three open-domain QA tasks in their experiments. The paper does not establish a universal RAG architecture or guarantee that retrieved context makes every generated claim factual.[32]

In an applied RAG pipeline, documents are selected, parsed, segmented, represented, and indexed. At query time the system may rewrite the query, retrieve candidates, rerank them, assemble context, and ask a generator to produce an answer. Each boundary can fail: parsing can omit text, segmentation can separate evidence, metadata filters can exclude the correct source, retrieval can miss it, truncation can drop it, and the generator can ignore or misstate it.[18][32]

Retrieval evaluation and answer evaluation should therefore be separated. Passage recall at a chosen depth asks whether known supporting evidence entered the candidate set. Ranking metrics ask where it appeared. Answer correctness, attribution, citation entailment, refusal behavior, and latency measure different downstream properties. Improving one does not mathematically imply improvement in the others.[9][32]

The retrievable unit must match the use case. Small passages can isolate an answer but lose context; large chunks preserve context but consume more of a generator's input and may dilute matching signals. Overlap can protect boundary-spanning evidence while creating near duplicates. Chunk size should be tested against annotated questions and source documents rather than copied as a universal character count.[1][32]

Access control is part of retrieval correctness. A system that retrieves a relevant passage for an unauthorized user has failed even if its ranking metric is high. Permissions should be enforced with semantics that cannot be bypassed by ANN approximation, caches, or a later generation step. Sensitive query and click logs also require retention, access, and audit policies.[13][23]

## Fairness, robustness, and failure analysis

Ranking allocates exposure. Singh and Joachims formalized fairness of exposure for position-biased rankings and showed how constraints can relate expected exposure to a relevance-based notion of merit. Their framework requires an explicit fairness definition and relevance estimates; it does not identify one fairness criterion suitable for all social or legal contexts.[33]

Aggregate effectiveness can hide unequal performance across languages, dialects, domains, or query intents. A useful audit stratifies results and reports uncertainty, while checking whether judgment depth and assessor expertise differ between slices. If one language has few topics, a large apparent gain may be statistically unstable.[29][31][33]

Robustness tests should include misspellings, rare identifiers, negation, temporal qualifiers, adversarial keyword stuffing, duplicates, and out-of-distribution topics. Lexical and dense systems fail differently. Dense similarity can return topically related but nonresponsive passages; lexical ranking can overvalue repeated exact terms; a cross-encoder can be sensitive to truncation and candidate ordering.[4][17]

Training data can introduce false negatives. ANCE used an asynchronously refreshed ANN index to mine negatives from the corpus while training a dense retriever, addressing the mismatch between random negatives and hard retrieval candidates. A mined "negative" may still be relevant when labels are incomplete, so negative selection and denoising are material parts of the method.[34]

Neural models can memorize benchmark-specific patterns or exploit artifacts. The broad neural-IR literature includes representation learning, interaction models, weak supervision, and multiple task definitions; results should be read with the training and evaluation protocol attached. "Semantic search" is not a guarantee of semantic understanding, factuality, or causal reasoning.[35]

Error analysis should sample both successful and failed queries, inspect false positives and false negatives, and distinguish index, retrieval, reranking, and presentation failures. Reviewing only low-scoring queries misses silent systematic errors, while reviewing only the top result misses recall failures deeper in the list.[1][11]

## Efficiency and operations

Retrieval latency is a distribution, not one mean. Reported service behavior should include percentile latency, concurrent load, collection size, index residency, result depth, filters, update rate, and hardware. A fast isolated ANN call does not include query encoding, network transit, authorization, reranking, or document fetching.[23][24]

Sparse indexes can be large because postings store identifiers, frequencies, positions, and skip data. Pibiri and Venturini survey techniques for compressing inverted indexes and the interactions among compression ratio, decoding speed, and query processing. Compression choices are workload-dependent; the smallest representation need not provide the lowest end-to-end latency.[36]

Dense systems have their own capacity equation: number of vectors, dimensions, bytes per component, graph or partition overhead, replicas, and metadata. Late-interaction systems multiply the vector count by retained document tokens. Quantization and pruning reduce cost but require an effectiveness and ANN-recall audit after conversion.[19][20][23]

Freshness creates another tradeoff. Batch-built indexes can be optimized globally but expose new documents late. Incremental updates improve freshness but can fragment structures, change corpus statistics, or leave stale vectors when an encoder changes. Deletions need verifiable propagation through replicas, caches, and derived indexes.[1][23]

Observability should separate empty results, timeouts, filter rejections, ANN misses, reranker failures, and downstream generation errors. Query logs need privacy controls, and sampled content may contain secrets or personal data. Monitoring labels should avoid recording raw sensitive text when aggregate counters or protected sampling can answer the operational question.[13]

## Reproducible system design

A retrieval experiment should begin with a written task definition: target users, corpus snapshot, retrievable unit, query source, relevance policy, cutoff, metric, and resource constraints. The test set should be held out before tuning. If judgments are pooled or incomplete, that limitation and the handling of unjudged documents should be explicit.[7][8][10]

For lexical retrieval, record analyzer versions, tokenization, normalization, field weights, BM25 formula and parameters, query operators, and expansion rules. For dense retrieval, record the exact checkpoint, pooling and normalization, embedding precision, distance function, ANN implementation and parameters, and index build procedure.[4][22][23]

For cascades, record the candidate source and depth at every stage, fusion formula, score normalization, truncation, reranker checkpoint, and final filtering. Evaluate first-stage recall as well as final ranking quality. This reveals whether a regression originated in candidate generation or later ordering.[15][17]

Compare against credible baselines. BM25 with a documented analyzer is more informative than an untuned lexical placeholder; exact vector search on a manageable subset helps quantify ANN loss; and a no-reranker run isolates the value and cost of reranking. Report per-dataset and per-slice results rather than only a macro-average.[24][27][30]

Before deployment, test permission enforcement, deletion propagation, stale indexes, overload behavior, and rollback. After deployment, monitor both system metrics and a stable relevance sample. Online experiments can measure user behavior, but click metrics require position-aware interpretation and guardrails for harmful or exclusionary changes.[13][14][33]

The central engineering principle is to preserve boundaries between claims. A benchmark result supports a conclusion about its protocol; an ANN benchmark supports a speed-recall point on specified hardware; a relevance judgment supports a defined information need; and a click supports an observed interaction under a presentation policy. Treating any of these as universal truth produces brittle retrieval systems and misleading documentation.[8][14][24]

## See also

- [BM25](https://aiwiki.ai/wiki/bm25)
- [TF-IDF](https://aiwiki.ai/wiki/tf_idf)
- [Embeddings](https://aiwiki.ai/wiki/embeddings)
- [Dense Passage Retrieval](https://aiwiki.ai/wiki/dense_passage_retrieval)
- [ColBERT](https://aiwiki.ai/wiki/colbert)
- [HNSW](https://aiwiki.ai/wiki/hnsw)
- [FAISS](https://aiwiki.ai/wiki/faiss)
- [Vector database](https://aiwiki.ai/wiki/vector_database)
- [Re-ranking](https://aiwiki.ai/wiki/re-ranking)
- [Benchmark](https://aiwiki.ai/wiki/benchmark)

## References

1. [Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schutze, *Introduction to Information Retrieval*, Cambridge University Press, 2008.](https://nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf)
2. [Gerard Salton, Anita Wong, and Chung-Shu Yang, "A Vector Space Model for Automatic Indexing," *Communications of the ACM* 18(11), 1975.](https://files.eric.ed.gov/fulltext/ED096986.pdf)
3. [Karen Sparck Jones, "A Statistical Interpretation of Term Specificity and Its Application in Retrieval," *Journal of Documentation* 28(1), 1972.](https://dmice.ohsu.edu/bedricks/courses/cs635_spring_2017/pdf/sparck_jones_1972.pdf)
4. [Stephen Robertson and Hugo Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond," *Foundations and Trends in Information Retrieval* 3(4), 2009.](https://www.ccs.neu.edu/home/vip/teach/IRcourse/IR_surveys/robertson_foundations.pdf)
5. [Sergey Brin and Lawrence Page, "The Anatomy of a Large-Scale Hypertextual Web Search Engine," *Computer Networks and ISDN Systems* 30, 1998.](https://web.stanford.edu/class/archive/cs/cs240/cs240.1046/readings/google.pdf)
6. [National Institute of Standards and Technology, "About the Text REtrieval Conference." ](https://trec.nist.gov/about.html)
7. [National Institute of Standards and Technology, "Relevance Judgments and Evaluation." ](https://trec.nist.gov/data/reljudge_eng.html)
8. [Chris Buckley, Darrin Dimmick, Ian Soboroff, and Ellen Voorhees, "Bias and the Limits of Pooling for Large Collections," *Information Retrieval* 10, 2007.](https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=51236)
9. [Kalervo Jarvelin and Jaana Kekalainen, "Cumulated Gain-Based Evaluation of IR Techniques," *ACM Transactions on Information Systems* 20(4), 2002.](https://trepo.tuni.fi/bitstream/handle/10024/65718/cumulated_gain_based_indicators_2002.pdf?isAllowed=y&sequence=1)
10. [Chris Buckley and Ellen M. Voorhees, "Retrieval Evaluation with Incomplete Information," *SIGIR 2004*.](https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=150469)
11. [Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees, "Overview of the TREC 2019 Deep Learning Track," NIST, 2020.](https://trec.nist.gov/pubs/trec28/papers/OVERVIEW.DL.pdf)
12. [Chris Burges et al., "Learning to Rank Using Gradient Descent," *ICML 2005*.](https://www.microsoft.com/en-us/research/wp-content/uploads/2005/08/icml_ranking.pdf)
13. [Thorsten Joachims et al., "Accurately Interpreting Clickthrough Data as Implicit Feedback," *SIGIR 2005*.](https://research.google/pubs/accurately-interpreting-clickthrough-data-as-implicit-feedback/)
14. [Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel, "Unbiased Learning-to-Rank with Biased Feedback," *WSDM 2017*.](https://www.cs.cornell.edu/people/tj/publications/joachims_etal_17a.pdf)
15. [Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher, "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods," *SIGIR 2009*.](https://research.google/pubs/reciprocal-rank-fusion-outperforms-condorcet-and-individual-rank-learning-methods/)
16. [Nils Reimers and Iryna Gurevych, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks," *EMNLP-IJCNLP 2019*.](https://aclanthology.org/D19-1410.pdf)
17. [Rodrigo Nogueira and Kyunghyun Cho, "Passage Re-ranking with BERT," 2019.](https://arxiv.org/pdf/1901.04085)
18. [Vladimir Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering," *EMNLP 2020*.](https://aclanthology.org/2020.emnlp-main.550.pdf)
19. [Omar Khattab and Matei Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT," *SIGIR 2020*.](https://arxiv.org/pdf/2004.12832)
20. [Keshav Santhanam et al., "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction," *NAACL 2022*.](https://arxiv.org/pdf/2112.01488)
21. [Thibault Formal, Benjamin Piwowarski, and Stephane Clinchant, "SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking," *SIGIR 2021*.](https://arxiv.org/pdf/2107.05720)
22. [Yu A. Malkov and Dmitry A. Yashunin, "Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs," *IEEE TPAMI* 42(4), 2020.](https://arxiv.org/pdf/1603.09320)
23. [Jeff Johnson, Matthijs Douze, and Herve Jegou, "Billion-Scale Similarity Search with GPUs," *IEEE Transactions on Big Data* 7(3), 2021.](https://arxiv.org/pdf/1702.08734)
24. [Martin Aumuller, Erik Bernhardsson, and Alexander Faithfull, "ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms," *Information Systems* 87, 2020.](https://arxiv.org/pdf/1807.05614)
25. [Payal Bajaj et al., "MS MARCO: A Human Generated MAchine Reading COmprehension Dataset," 2016.](https://arxiv.org/pdf/1611.09268)
26. [Microsoft, "MS MARCO Datasets." ](https://microsoft.github.io/msmarco/Datasets.html)
27. [Nandan Thakur et al., "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models," *NeurIPS Datasets and Benchmarks 2021*.](https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/65b9eea6e1cc6bb9f0cd2a47751a186f-Paper-round2.pdf)
28. [Niklas Muennighoff et al., "MTEB: Massive Text Embedding Benchmark," *EACL 2023*.](https://aclanthology.org/2023.eacl-main.148.pdf)
29. [Kenneth Enevoldsen et al., "MMTEB: Massive Multilingual Text Embedding Benchmark," *ICLR 2025*.](https://arxiv.org/pdf/2502.13595)
30. [Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin, "Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval," *MRL 2021*.](https://aclanthology.org/2021.mrl-1.12.pdf)
31. [Xinyu Zhang et al., "MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages," *Transactions of the Association for Computational Linguistics* 11, 2023.](https://aclanthology.org/2023.tacl-1.63.pdf)
32. [Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," *NeurIPS 2020*.](https://papers.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf)
33. [Ashudeep Singh and Thorsten Joachims, "Fairness of Exposure in Rankings," *KDD 2018*.](https://www.cs.cornell.edu/~tj/publications/singh_joachims_18a.pdf)
34. [Lee Xiong et al., "Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval," *ICLR 2021*.](https://arxiv.org/pdf/2007.00808)
35. [Bhaskar Mitra and Nick Craswell, "An Introduction to Neural Information Retrieval," *Foundations and Trends in Information Retrieval* 13(1), 2018.](https://www.microsoft.com/en-us/research/uploads/prod/2017/06/fntir2018-neuralir-mitra.pdf)
36. [Giulio Ermanno Pibiri and Rossano Venturini, "Techniques for Inverted Index Compression," *ACM Computing Surveys* 53(6), 2021.](https://jermp.github.io/assets/pdf/papers/CSUR2021.pdf)