Question answering

Question answering (QA) is the task of producing an answer to a question, usually expressed in natural language. In its information-retrieval form, the defining difference is the output: a QA system attempts to return an answer, while a conventional search system returns a ranked set of documents that may contain one.^[1] The field sits across natural language processing, information retrieval, and artificial intelligence.

QA is not one fixed benchmark or architecture. A question can be paired with a passage, searched against a large corpus, evaluated over a table or knowledge graph, grounded in an image, or answered only from a model's parameters. The expected output can be a copied text span, a database value, a choice, a short generated answer, or a longer answer with evidence. These dimensions overlap. For example, an open-domain system can retrieve passages and then either extract a span or generate a response from them.

A useful QA evaluation therefore asks more than whether the final string resembles a reference answer. Depending on the task, it may also test whether the source contained an answer, whether the correct evidence was retrieved, whether multiple pieces of evidence were combined, whether the system abstained when evidence was insufficient, and whether every statement in a generated answer is supported.

Scope and task dimensions

Question-answering tasks can be described along four independent axes.

Scroll sideways for more →

The labels used in the literature describe different points on these axes rather than mutually exclusive categories:

• Reading comprehension supplies a passage or document with the question. In extractive reading comprehension, the answer is a contiguous span in that context.
• Open-domain QA does not supply a gold passage. The system must search a large collection before selecting or generating an answer.
• Closed-book QA prohibits retrieval at inference time and tests what a model can produce from its learned parameters.
• Multi-hop QA requires evidence from more than one sentence, passage, or document.
• Conversational QA places the question in a dialogue, so the current turn may depend on earlier questions and answers.
• Table QA operates over rows, columns, cells, and sometimes aggregation operations.
• Knowledge-graph QA maps a question to entities, relations, or executable graph queries.
• Visual question answering grounds the answer in an image. Document QA can combine text, layout, tables, and visual content.

Separate index pages catalog question-answering models, document QA models, table QA models, and visual QA models. This article concerns the tasks, methods, datasets, and evaluation principles rather than a current model ranking.

History

Domain-specific systems

Early QA systems worked in narrow, explicitly represented domains. BASEBALL, described in 1961, accepted ordinary-English questions about stored American League game records for one year. Its database held such fields as month, day, place, teams, and scores. The original system also restricted question structure, including dependent clauses, logical connectives, and some superlatives, so its broad-sounding interface remained tied to a small vocabulary and a carefully controlled database.^[2]

LUNAR was built to let lunar geologists query chemical-analysis data from the Apollo missions without learning the database's internal representation. A 1971 demonstration of a preliminary version produced a more qualified result than is sometimes repeated: 78 percent of questions were understood and answered correctly, 12 percent failed because of clerical problems in the not-yet-debugged system, and 10 percent failed because of substantial parsing or semantic-interpretation problems. The test excluded requests outside the database and some comparative constructions.^[3]

These systems established enduring parts of the problem: interpreting a question, mapping its meaning to a data source, executing a search or computation, and expressing the result. Their rules and vocabularies were engineered for one domain, however, so success did not imply general language understanding.

Shared evaluation

The Text REtrieval Conference created a question-answering track in 1999. Its official task description contrasted QA with document retrieval: participating systems were expected to return short answers to fact-based, open-domain questions rather than a ranked document list.^[1] Shared questions, collections, judgments, and scoring made it possible to compare systems under a common protocol and helped establish open-domain QA as a distinct retrieval problem.

The publication of large reading-comprehension datasets later shifted much of the field toward supervised machine learning. The Stanford Question Answering Dataset (SQuAD) paired crowd-written questions with Wikipedia passages and span answers. Its original release contained more than 100,000 questions from more than 500 articles.^[4][5] SQuAD 2.0 added more than 50,000 adversarial unanswerable questions, requiring a model to decide whether a passage supports any answer before extracting one.^[6]

Neural architectures such as BiDAF and then pretrained language models made passage QA a prominent test of representation learning. BiDAF kept a sequence of context representations instead of compressing the passage into one vector and used attention in both question-to-context and context-to-question directions.^[7] BERT showed that a pretrained bidirectional Transformer encoder could be adapted to extractive QA with only an additional start-position and end-position output layer. The BERT paper reported test F1 scores of 93.2 on SQuAD 1.1 and 83.1 on SQuAD 2.0 for its submitted systems.^[8] Those are version-specific historical benchmark results, not evidence that all QA problems were solved.

Passage and extractive QA

In extractive QA, the input normally consists of a question and a passage. The answer is represented by the start and end positions of a span within that passage. If a passage says "The treaty was signed in Paris in 1951" and the question asks where it was signed, a span model can return "Paris" without generating new wording.

This formulation provides several advantages. Training targets are exact token locations, inference can be reduced to scoring candidate boundaries, and the returned answer is visibly present in the supplied evidence. It also imposes a strict limitation: a correct response must appear as a contiguous span. A system cannot naturally combine two distant statements, normalize a value, explain an answer, or say the same thing in substantially different words unless the benchmark introduces a separate mechanism.

SQuAD and answerability

SQuAD's questions were written after annotators had read passages from Wikipedia, so its task is passage-conditioned reading rather than open-domain search. The original release contained more than 100,000 question-answer pairs on more than 500 articles and reported human and model performance with Exact Match and token-level F1.^[4][5] Because the first release only contained answerable questions, a model could assume that some passage span was correct.

SQuAD 2.0 deliberately removed that assumption. Its added questions were written to look relevant to a supplied paragraph while remaining unanswerable from it. The model therefore had to compare the best answer-span score with a no-answer alternative, and evaluation included the decision to abstain.^[6] This distinction is operationally important: answer extraction and answerability detection are related but different sources of error.

The official SQuAD explorer maintains separate leaderboards and evaluation scripts for the two versions. It reports that SQuAD 2.0 combines the original answerable questions with more than 50,000 unanswerable questions and exposes both Exact Match and F1 results.^[4] Leaderboard values should always be attached to a named dataset version and evaluation protocol because preprocessing, ensembling, external data, and submission rules can differ.

Attention-based readers

BiDAF represented each context word with character, word, and contextual features, then calculated an attention flow between the question and passage. Its context-to-question component identified question words most relevant to each passage position, while its question-to-context component highlighted passage positions with strong similarity to some question word. The resulting sequence remained available to later modeling and output layers.^[7]

The architecture is historically important because it illustrates a reader built specifically for machine comprehension. It predates the broad use of large pretrained encoders and performs joint question-passage reasoning through a task-specific stack. Later pretrained systems moved much of that representation learning into a general model and used a smaller task head.

The BERT paper emphasized that the pretrained encoder could be fine-tuned for QA with one additional output layer and without substantial task-specific architectural changes.^[8] The resulting QA system still followed the benchmark's extractive constraint even though BERT's pretraining was not specific to question answering.

What passage scores do not measure

High performance on a passage benchmark does not test corpus retrieval when the correct passage is already supplied. It also does not necessarily test whether a system can resist misleading but topically related documents, cite evidence, reconcile conflicting sources, or answer questions whose wording and domains differ from the training set.

The MRQA 2019 shared task addressed part of this generalization problem by converting 18 existing QA datasets into a common format and evaluating systems on held-out datasets. The best shared-task system averaged 72.5 F1 on the test datasets, 10.7 points above the organizers' BERT baseline.^[9] The setup made cross-dataset transfer, rather than optimization for one benchmark, the target of comparison.

Open-domain QA

Open-domain QA adds a search problem. A system receives a question but not the evidence passage, and must locate relevant material in a large collection before answering. A typical pipeline has a retriever, which selects candidate passages, and a reader or generator, which turns those passages into an answer.

This decomposition creates at least two distinct failure modes:

1. Retrieval failure: the needed evidence is absent from the retrieved set.
2. Answering failure: sufficient evidence is retrieved, but the reader selects, combines, or expresses it incorrectly.

An end-to-end answer score mixes these failures. Retriever recall at a fixed cutoff and reader performance with gold versus retrieved passages help locate the source of an error.

Sparse retrieval and DrQA

DrQA was an early neural open-domain system evaluated over Wikipedia. Its document retriever used hashed bigram features and TF-IDF matching, while its document reader used a recurrent neural network to extract answers from candidate text.^[10] The design separated inexpensive corpus-wide matching from more expensive contextual reading.

TF-IDF retrieval depends on lexical overlap. It can work well when the question and relevant passage share distinctive terms, but may miss paraphrases or semantically related wording. Query expansion, entity handling, and better document segmentation can improve a sparse system, but the underlying representation remains tied to observed terms.

Natural Questions

Natural Questions was designed around real information-seeking queries rather than questions written while viewing a passage. Its authors released 307,373 training examples, 7,830 development examples, and 7,842 test examples. The questions were anonymized, aggregated queries issued to Google, and annotators viewed a Wikipedia page drawn from the top search results. They marked a long answer, one or more short answers, or no answer when the page did not support one.^[11]

This annotation structure separates several decisions. A system may need to identify the relevant region of a long page, select a concise answer within that region, and detect when the chosen page has no answer. The open version used by some research removes the supplied page and makes retrieval from a corpus part of the task; results from those settings should not be mixed without stating the protocol.

Dense passage retrieval

Dense Passage Retrieval (DPR) replaced lexical matching with two learned encoders: one for questions and one for passages. Retrieval used similarity between their vector representations. On the open-domain datasets studied in the paper, the authors reported gains of 9 to 19 percentage points over a strong BM25 system in top-20 passage retrieval accuracy.^[12] That result is specific to the paper's datasets, indexes, and definition of a matching passage.

Dense and sparse methods make different errors. Dense retrieval can connect paraphrases with little word overlap, while sparse retrieval can precisely match rare names, identifiers, and quoted phrases. Many practical designs therefore compare, combine, or rerank candidates from more than one retriever. The appropriate choice depends on the corpus, latency constraints, update frequency, and the types of questions expected.

Retrieval-augmented generation

Retrieval-augmented generation (RAG) couples retrieved evidence to a generative model. The 2020 RAG paper combined a sequence-to-sequence model's parametric memory with a dense index of Wikipedia as nonparametric memory. It defined RAG-Sequence, in which one retrieved document supports the generated sequence, and RAG-Token, in which the latent document can vary by output token. The authors reported state-of-the-art results at publication on three open-domain QA tasks.^[13]

RAG does not guarantee a grounded answer. Retrieval may return irrelevant or outdated text, relevant evidence may be truncated, and a generator may add unsupported material even when the correct source is present. The architecture makes external evidence available; faithfulness still has to be measured.

Fusion-in-Decoder (FiD) couples passage retrieval with a sequence-to-sequence answer model. The paper found that performance improved as the number of retrieved passages increased and reported state-of-the-art results on Natural Questions and TriviaQA at publication.^[14] The result illustrates that a generator can aggregate evidence from several retrieved passages, while remaining dependent on the quality of those passages.

Generative and closed-book QA

Generative QA produces an answer token by token rather than selecting one contiguous source span. This permits paraphrase, normalization, synthesis, and explanation. It also removes the automatic guarantee that the answer text occurs in the evidence.

The T5 study cast all of its tasks, including QA, into a text-to-text format. Inputs and outputs were text strings, allowing one encoder-decoder framework and training objective to cover different NLP problems.^[15] UnifiedQA applied this idea across 19 QA datasets and four answer formats. Its paper reported that one model trained on the combined data performed comparably to eight separately trained models and transferred to 12 unseen QA datasets, reaching the best reported result on 10 of them at the time.^[16]

These results support transfer across QA formats, not the claim that all QA datasets are equivalent. A multiple-choice science question, an extractive Wikipedia question, and a commonsense yes-or-no question can share an input-output interface while requiring different evidence and evaluation assumptions.

Closed-book evaluation

Closed-book QA asks a model to answer without access to documents or another external source at inference time. A 2020 study fine-tuned T5 for open-domain questions while providing no retrieved context and found that performance increased with model size; the largest model was competitive with some retrieval-based systems of that period.^[17] This setting probes information encoded in parameters, but a correct answer does not expose where the information came from or whether it remains current.

Closed-book scores can also be affected by overlap between pretraining material and evaluation questions, memorization of answer patterns, and benchmark popularity. For these reasons, closed-book and retrieval-grounded results answer different research questions. The former studies parametric recall and generalization; the latter also studies evidence access, provenance, and source-conditioned reasoning.

Answer generation is not evidence

A fluent answer can be wrong. Generation quality, factual correctness, and source support are separate properties. Even when a system returns citations, the citations may not entail every claim, may point only to topically related text, or may omit important qualifiers. QA interfaces should therefore avoid treating fluency, length, or citation presence as proof of correctness.

This distinction becomes especially important for long answers. Exact string matching is unsuitable when several phrasings can be correct, but unconstrained model-based judging can introduce its own errors. Evaluation may need atomic claim checks, evidence entailment, citation coverage, and human review in addition to semantic similarity.

Multi-hop and conversational QA

Multi-hop reasoning

Multi-hop questions require evidence to be combined. One document may identify a person, while another provides a date associated with that person. A comparison question may require retrieving the same attribute for two entities before comparing the values. If the benchmark supplies all candidate passages, it tests evidence selection and reasoning; if it requires corpus search, it also tests multi-step retrieval.

HotpotQA contains about 113,000 Wikipedia-based question-answer pairs. Its questions were designed to use multiple documents, and it provides sentence-level supporting-fact annotations. It also includes comparison questions. Those features allow separate scoring of the answer and the evidence sentences used to justify it.^[18]

Supporting-fact labels make an important error visible: a system can sometimes guess the correct short answer from a shortcut or from one incomplete passage. Joint answer-and-evidence evaluation rewards models that identify the intended support as well. It does not by itself prove that the system performed the human-interpretable reasoning suggested by the evidence chain.

Conversational context

Conversational QA adds dialogue history. A follow-up such as "When did that happen?" cannot be interpreted without resolving "that" from previous turns. Later questions can also narrow a topic, switch entities, challenge an earlier answer, or ask for clarification.

CoQA contains 127,000 questions in 8,000 conversations over passages from seven domains. Its answers are free-form text paired with evidence rationales. The paper's best system scored 65.4 F1 compared with human performance of 88.8, illustrating the gap under that release's protocol.^[19]

QuAC contains 14,000 information-seeking dialogues and about 100,000 questions. In its collection setup, a student asked questions without seeing the hidden Wikipedia section, while a teacher answered from the text. This asymmetry encouraged exploratory questions rather than questions composed directly from visible answer sentences. The best model reported in the paper remained 20 F1 points below human performance.^[20]

ChatRAG Bench later combined ten datasets to evaluate conversational QA with retrieval and long-context input. In the ChatQA paper, the authors reported an average score of 54.14 for their 70-billion-parameter ChatQA 1.0 model, compared with 53.90 for GPT-4-0613 and 54.03 for GPT-4 Turbo under their benchmark and evaluation setup.^[21] This narrow comparison should not be generalized to other versions, tasks, judges, or deployment conditions.

Structured and multimodal QA

Tables

Table QA requires a model to connect language with rows, columns, cells, and sometimes operations such as counting or summing. Answers can be directly selected cells, computed values, or generated text. A table's structure carries meaning that is lost if cells are treated as an unordered bag of words.

WikiTableQuestions introduced 22,033 complex questions over tables taken from Wikipedia. The questions include compositional operations, and the original work used question-answer supervision rather than annotated logical forms.^[22] Because several programs can produce the same answer, learning from denotations creates an additional ambiguity: an executable program can arrive at the correct value for the wrong reason.

TAPAS extended BERT-style encoding to tables. It predicted selected cells and, when needed, an aggregation operator without generating an explicit logical form. The paper reported an increase on the Sequential Question Answering dataset from 55.1 to 67.2 and competitive results on WikiTableQuestions and WikiSQL.^[23] These figures refer to the authors' evaluation settings and do not compare every later table-QA system.

Text-and-table QA can be harder than either modality alone. The relevant row may be identified in prose, a number may have to be read from a table, and the final answer may require arithmetic. Evaluation should therefore distinguish retrieval of the right document or table from execution of the right operation.

Images and documents

Visual QA asks a natural-language question about an image. The original VQA dataset paired roughly 250,000 images with about 760,000 questions and 10 million human answers.^[24] Questions included object, attribute, count, activity, and scene queries, and the task used open-ended answers rather than a fixed label set alone.

Image QA does not reduce to object recognition. A model may need to read text in the scene, resolve spatial relationships, count instances, infer an activity, or distinguish what is visible from what is merely plausible. Dataset biases can permit plausible answers without adequate image use, so controlled comparisons and counterexamples are important.

Document QA broadens the evidence further. A page can contain prose, headings, forms, tables, diagrams, and layout cues. Systems may combine optical character recognition, layout representations, visual encoders, retrieval, and answer generation. A document benchmark should state whether text transcription is provided, whether answer evidence can cross pages, and whether the answer is extractive or generated.

Knowledge graphs

Knowledge-graph QA resolves questions against structured entities and relations. A semantic parser may translate a question into a graph query, while other systems rank candidate paths or combine graph and text retrieval.

WebQuestions studied learning a semantic parser for Freebase from question-answer pairs rather than gold logical forms.^[25] WebQuestionsSP later supplied full semantic parses for 4,737 questions and partial annotations for another 1,073, according to Microsoft's dataset release.^[26] These resources support evaluation of whether a system maps language to the intended structure, not only whether it reaches a matching answer.

GrailQA contains 64,331 questions and evaluates three generalization settings: i.i.d., compositional, and zero-shot. Its zero-shot split tests unseen schema items, while the compositional split tests novel combinations of seen components.^[27] Separating these settings helps show whether a graph-QA model is matching familiar templates or can generalize its parsing decisions.

Benchmark design

No single benchmark represents question answering as a whole. The following datasets illustrate different evidence and output conditions.

Scroll sideways for more →

The counts above come from the respective dataset papers or official releases.^{[5][6][11][18][19][20][22][24][27]} They should not be silently combined with later versions, filtered subsets, or leaderboards that use different test sets.

Collection effects

How questions are collected shapes what a benchmark measures. Questions written after reading a paragraph tend to share its vocabulary and presuppose that an answer is present. Search queries collected before annotators inspect evidence can be shorter, ambiguous, and sometimes unanswerable from the selected page. Questions in a conversation contain references to earlier turns. Questions written for a table can assume operations over rows or columns.

These differences can create shortcuts. A model may exploit annotation patterns, common answer types, or repeated templates rather than the intended reasoning. Adversarial examples, held-out domains, counterfactual inputs, and compositional or zero-shot splits address different shortcuts, but none removes the need to inspect errors.

Ambiguity

Some questions have multiple defensible interpretations. AmbigQA identified this problem in open-domain QA and annotated 14,042 questions from Natural Questions Open. More than half were found to be ambiguous, and the task required returning all plausible answers together with disambiguated question rewrites.^[28]

Ambiguity should not be confused with model uncertainty. A system can be confident while selecting only one of several valid readings. Conversely, a question may be unambiguous but lack sufficient evidence in the available corpus. Evaluation and user interfaces benefit from distinguishing alternative interpretations, missing evidence, and low model confidence.

Evaluation

Exact Match and token F1

Exact Match (EM) assigns full credit when a normalized prediction exactly equals an accepted reference answer and zero otherwise. SQuAD's normalization convention lowercases text and removes punctuation, articles, and extra whitespace before comparison.^[4] EM is transparent but strict: a correct explanatory phrase can fail when the reference is a shorter span.

Token F1 treats prediction and reference as bags of tokens. Precision is the fraction of predicted tokens that overlap the reference, recall is the fraction of reference tokens recovered, and their harmonic mean is the F1 score. With multiple references, implementations commonly take the maximum score over references. The exact tokenization and normalization rules are part of the metric and must be kept fixed when comparing systems.

EM and F1 score work naturally for short extractive answers. They are less reliable for longer generated responses because a correct paraphrase can have little token overlap, while an answer with high overlap can still introduce a consequential false statement.

Retrieval metrics

Open-domain systems also evaluate retrieval. Recall at k asks whether the first k retrieved items include sufficient evidence or an answer-bearing passage. Mean reciprocal rank rewards placing the first relevant item early. These metrics require a definition of relevance: string containment, annotated evidence, document identity, or human judgment can produce different results.

Answer-string containment is convenient but imperfect. A passage can contain the answer phrase in an unrelated context, and a correct evidence passage may express the answer differently. Retrieval evaluation is strongest when evidence annotations match the task's actual support requirement.

Evidence and reasoning metrics

HotpotQA scores supporting facts as well as answers.^[18] Similar designs can evaluate document selection, passage selection, and sentence-level evidence. For a multi-hop question, evidence recall shows whether every required step was available, while evidence precision penalizes irrelevant support.

These labels remain dataset-specific. A benchmark's annotated path may be one valid explanation rather than the only possible path. Models can also produce the right evidence without executing a sound reasoning process. Evidence metrics are therefore a useful diagnostic, not a direct measurement of internal reasoning.

Generated answers and citations

Generated answers call for at least three distinct judgments:

1. Correctness: Is the answer to the question right?
2. Faithfulness: Is each claim supported by the supplied or retrieved evidence?
3. Completeness: Does the response include the information needed to answer the question?

Citation quality adds further questions: does each citation point to evidence that entails the attached claim, and are all material claims covered? The ALCE benchmark evaluated retrieval and citation generation end to end. Its authors reported that, for their ELI5 setting, even the best systems lacked complete citation support for about half of their claims.^[29] This result concerns the evaluated models and benchmark, but it demonstrates why visible citations alone are not a sufficient metric.

Instruction-following answers introduce another measurement problem. A 2024 study found that verbose model outputs make conventional EM and F1 unreliable and proposed evaluating both correctness and faithfulness to knowledge. It also found that the evaluated models could hallucinate and had difficulty judging whether retrieved evidence was relevant.^[30] Long-form QA needs scoring rules that account for unsupported additions, not only whether a reference answer appears somewhere in the response.

Reliability and limitations

Hallucination and source conflict

A retrieved context can reduce dependence on parametric memory, but it cannot by itself prevent hallucination. RAGTruth contains nearly 18,000 naturally generated responses from retrieval-augmented systems, annotated for word-level hallucinations across QA, data-to-text, and summarization tasks.^[31] The benchmark explicitly includes unsupported and contradictory content produced despite retrieved source material.

Conflicting sources create a different problem. A system may retrieve documents from different dates, jurisdictions, experimental settings, or reliability levels. Simply combining their statements can produce an incoherent answer. A robust response should preserve dates and qualifications, prefer authoritative evidence for the claim, and state when reliable sources disagree rather than inventing a consensus.

Truthfulness

TruthfulQA contains 817 questions in 38 categories designed to elicit false answers that echo common misconceptions. In the original evaluation, the best tested model was truthful on 58 percent of questions, compared with 94 percent for humans.^[32] The paper found that scaling alone did not reliably improve truthfulness under its setup.

Truthfulness differs from ordinary benchmark accuracy. A model may reproduce a widely repeated misconception because it is statistically common, while a truthfulness test rewards rejecting that premise. This motivates adversarial question design and source-based verification, especially for health, law, finance, and other high-stakes domains.

Long context

More context is not automatically better. "Lost in the Middle" evaluated multi-document QA and synthetic key-value retrieval while changing the position of relevant information. The authors found that performance was often strongest when relevant evidence occurred near the beginning or end of the input and degraded when it appeared in the middle.^[33]

Long-context QA can therefore fail even when the answer-bearing passage is technically inside the model's input window. Retrieval, ordering, chunking, deduplication, and context compression remain relevant engineering choices. Evaluations should vary evidence position and include distractors rather than infer reliable use from maximum context length alone.

Distribution shift

MRQA's held-out datasets showed that performance can drop when a reader moves across domains and collection procedures.^[9] GrailQA separately tests new combinations and unseen knowledge-graph schema items.^[27] These are different forms of distribution shift, and a single in-domain score does not predict both.

Real deployments add more shifts: new document templates, changed terminology, multilingual questions, time-sensitive facts, OCR errors, and user questions outside the intended domain. Monitoring should sample actual failures and re-evaluate retrieval, answerability, and evidence faithfulness after corpus or model changes.

Abstention and uncertainty

A QA system should be able to state that the available evidence is insufficient. SQuAD 2.0 measures no-answer decisions within a supplied passage,^[6] while Natural Questions permits a null annotation when the selected page lacks an answer.^[11] Open-domain abstention is harder because failure to find evidence does not prove that no evidence exists.

Confidence scores are not automatically calibrated probabilities. A useful abstention policy must be evaluated against the cost of wrong answers and missed answers for the specific application. Selective prediction curves, calibration checks, and human escalation policies can be more informative than one universal threshold.

Current research directions

Recent benchmarks increasingly evaluate complete retrieval-and-answering systems rather than an isolated reader. FRAMES, published in 2025, combines factuality, retrieval, and reasoning in an end-to-end RAG evaluation with multi-hop questions. The paper's baseline rose from 0.40 accuracy without retrieval to 0.66 with its multi-step retrieval setup, leaving substantial room under that protocol.^[34]

T2-RAGBench, published in 2026, targets questions that combine text and tables. It contains 23,088 question-context-answer triples and was designed to test both retrieval and numerical reasoning with context-independent questions.^[35] The benchmark reflects a broader move toward mixed evidence, where locating a table and executing the required operation are both part of the answer.

Other active directions include better evidence attribution, robust retrieval under corpus change, multilingual and cross-lingual QA, efficient use of long documents, calibration, and evaluation of agents that perform iterative search. Results in these areas should be reported with exact model versions, corpus snapshots, tool access, and evaluation procedures because changes in any of those components can alter the task.

Applications and system design

QA is used wherever users need a direct response from a bounded source: a document collection, product manual, policy library, scientific corpus, database, table, or set of images. The appropriate architecture follows from the evidence and risk, not from the label "question answering."

A passage-level tool can use an extractive reader when concise source spans are sufficient. A large changing corpus usually requires retrieval and may add reranking before reading or generation. Tables may require explicit aggregation. A long answer may require claim-level citation checks. A high-stakes setting may require abstention and human review even when automatic benchmark scores are strong.

A practical QA pipeline can be audited as a sequence:

1. Interpret the request. Resolve dialogue references, expected answer type, and any constraints.
2. Select evidence sources. Define which collections, databases, tables, or images are authoritative and current.
3. Retrieve candidates. Measure whether relevant evidence appears within the material passed downstream.
4. Read or execute. Extract a span, execute a structured operation, or generate from the evidence.
5. Verify support. Check that the answer and each material generated claim are entailed by the cited source.
6. Abstain or escalate. Do not convert missing, conflicting, or inadequate evidence into a confident answer.
7. Log the basis. Retain source identifiers, corpus versions, model versions, and evaluation outcomes needed for review.

The same sequence exposes where improvement is needed. If retrieval recall is low, a stronger generator cannot recover missing evidence. If evidence is present but the answer is wrong, reader or reasoning changes are relevant. If the answer is correct but unsupported additions appear, generation constraints and faithfulness checks matter. Separating these stages makes QA quality easier to measure than a single end-to-end score alone.

References

1. ^National Institute of Standards and Technology, "Question Answering Track," Text REtrieval Conference trec.nist.gov/...qa
2. ^Bert F. Green Jr., Alice K. Wolf, Carol Chomsky, and Kenneth Laughery, "BASEBALL: An Automatic Question-Answerer," Proceedings of the Western Joint Computer Conference, 1961 web.stanford.edu/...p219-green.pdf
3. ^William A. Woods, "Natural Language Question Answering," Advances in Computers, volume 17, 1978, doi:10.1016/S0065-2458(08)60390-3 web.stanford.edu/...woods.pdf
4. ^Stanford NLP Group, "The Stanford Question Answering Dataset" rajpurkar.github.io/SQuAD-explorer
5. ^Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, "SQuAD: 100,000+ Questions for Machine Comprehension of Text," EMNLP 2016 aclanthology.org/D16-1264
6. ^Pranav Rajpurkar, Robin Jia, and Percy Liang, "Know What You Don't Know: Unanswerable Questions for SQuAD," ACL 2018 aclanthology.org/P18-2124
7. ^Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi, "Bidirectional Attention Flow for Machine Comprehension," ICLR 2017 arxiv.org/...1611.01603
8. ^Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," NAACL 2019 aclanthology.org/N19-1423
9. ^Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen, "MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension," EMNLP-IJCNLP 2019 aclanthology.org/D19-5801
10. ^Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes, "Reading Wikipedia to Answer Open-Domain Questions," ACL 2017 aclanthology.org/P17-1171
11. ^Tom Kwiatkowski et al., "Natural Questions: A Benchmark for Question Answering Research," Transactions of the Association for Computational Linguistics, volume 7, 2019 aclanthology.org/Q19-1026
12. ^Vladimir Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering," EMNLP 2020 aclanthology.org/2020.emnlp-main.550
13. ^Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS 2020 proceedings.neurips.cc/...bc26945df7481e5-Abstract
14. ^Gautier Izacard and Edouard Grave, "Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering," EACL 2021 aclanthology.org/2021.eacl-main.74
15. ^Colin Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer," Journal of Machine Learning Research, volume 21, 2020 jmlr.org/...20-074
16. ^Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi, "UNIFIEDQA: Crossing Format Boundaries With a Single QA System," Findings of EMNLP 2020 aclanthology.org/2020.findings-emnlp.171
17. ^Adam Roberts, Colin Raffel, and Noam Shazeer, "How Much Knowledge Can You Pack Into the Parameters of a Language Model?," EMNLP 2020 aclanthology.org/2020.emnlp-main.437
18. ^Zhilin Yang et al., "HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering," EMNLP 2018 aclanthology.org/D18-1259
19. ^Siva Reddy, Danqi Chen, and Christopher D. Manning, "CoQA: A Conversational Question Answering Challenge," Transactions of the Association for Computational Linguistics, volume 7, 2019 aclanthology.org/Q19-1016
20. ^Eunsol Choi et al., "QuAC: Question Answering in Context," EMNLP 2018 aclanthology.org/D18-1241
21. ^Zihan Liu et al., "ChatQA: Surpassing GPT-4 on Conversational QA and RAG," NeurIPS 2024 proceedings.neurips.cc/...eca7d591e374b9d-Abstract
22. ^Panupong Pasupat and Percy Liang, "Compositional Semantic Parsing on Semi-Structured Tables," ACL-IJCNLP 2015 arxiv.org/...1508.00305
23. ^Jonathan Herzig et al., "TaPas: Weakly Supervised Table Parsing via Pre-training," ACL 2020 aclanthology.org/2020.acl-main.398
24. ^Stanislaw Antol et al., "VQA: Visual Question Answering," ICCV 2015 openaccess.thecvf.com/..._Question_ICCV_2015_paper
25. ^Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang, "Semantic Parsing on Freebase from Question-Answer Pairs," EMNLP 2013 aclanthology.org/D13-1160
26. ^Microsoft Research, "WebQuestionsSP" microsoft.com/...details
27. ^Yu Gu et al., "Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases," The Web Conference 2021 arxiv.org/...2011.07743
28. ^Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer, "AmbigQA: Answering Ambiguous Open-domain Questions," EMNLP 2020 aclanthology.org/2020.emnlp-main.466
29. ^Tianyu Gao et al., "Enabling Large Language Models to Generate Text with Citations," EMNLP 2023 aclanthology.org/2023.emnlp-main.398
30. ^Vaibhav Adlakha et al., "Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering," Transactions of the Association for Computational Linguistics, volume 12, 2024 aclanthology.org/2024.tacl-1.38
31. ^Cheng Niu et al., "RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models," ACL 2024 aclanthology.org/2024.acl-long.585
32. ^Stephanie Lin, Jacob Hilton, and Owain Evans, "TruthfulQA: Measuring How Models Mimic Human Falsehoods," ACL 2022 aclanthology.org/2022.acl-long.229
33. ^Nelson F. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," Transactions of the Association for Computational Linguistics, volume 12, 2024 aclanthology.org/2024.tacl-1.9
34. ^Satyapriya Krishna et al., "Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation," NAACL 2025 aclanthology.org/2025.naacl-long.243
35. ^Jan Strich et al., "T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation," EACL 2026 aclanthology.org/2026.eacl-long.8