# SimpleQA

> Source: https://aiwiki.ai/wiki/simpleqa
> Updated: 2026-06-23
> Categories: AI Benchmarks, AI Safety, Natural Language Processing
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

**SimpleQA** is a factuality [benchmark](/wiki/benchmark) released by [OpenAI](/wiki/openai) on October 30, 2024 that measures whether [large language models](/wiki/llm) can answer short, fact-seeking questions correctly instead of producing [hallucinations](/wiki/hallucination).[^1] It consists of 4,326 questions, each adversarially collected to be hard and each written so that, in the authors' words, "there exists only a single, indisputable answer," verified through a two-stage human annotation process.[^2] Every model response is graded into one of three categories, correct, incorrect, or not attempted, which lets SimpleQA measure not only accuracy but also calibration: whether a model knows what it does not know.[^2] Frontier models score surprisingly low: in the original paper no model exceeded 50%, with GPT-4o reaching 38.2% and OpenAI o1-preview leading at 42.7%.[^2]

| SimpleQA |
| --- |
| Overview |
| Full name | SimpleQA: Measuring Short-Form Factuality in Large Language Models |
| Abbreviation | SimpleQA |
| Description | A factuality benchmark measuring language models' ability to answer short, fact-seeking questions accurately without hallucination |
| Release date | 2024-10-30 |
| Latest version | 1.0 |
| Benchmark updated | 2024-11 |
| Authors | Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus |
| Organization | [OpenAI](/wiki/openai) |
| Technical Details |
| Type | Factuality, Question Answering, Hallucination Detection |
| Modality | Text |
| Task format | Short-form question answering |
| Number of tasks | Multiple topic domains |
| Total examples | 4,326 questions |
| Evaluation metric | [Accuracy](/wiki/accuracy), F-score, Not Attempted rate |
| Domains | Science & Technology, Politics, Art, History, Entertainment, Geography |
| Languages | English |
| Performance |
| Human performance | Not explicitly measured |
| Baseline | 8.6% (GPT-4o-mini) |
| SOTA score | 62.5% (parametric, original) |
| SOTA model | GPT-4.5 |
| SOTA date | 2025-02 |
| Saturated | No (parametric); see notes on retrieval-augmented scores |
| Resources |
| Website | [Official website](https://openai.com/index/introducing-simpleqa/) |
| Paper | [Paper](https://arxiv.org/abs/2411.04368) |
| GitHub | [Repository](https://github.com/openai/simple-evals) |
| Dataset | [Download](https://huggingface.co/datasets/basicv8vc/SimpleQA) |
| License | MIT |

SimpleQA was announced in an OpenAI blog post by Jason Wei and colleagues on October 30, 2024[^1], with the accompanying paper submitted to arXiv on November 7, 2024[^2]. OpenAI describes it as "a simple, targeted evaluation for whether models know what they know."[^2] The benchmark was designed to be challenging (adversarially collected against [GPT-4](/wiki/gpt-4) responses), easy to grade (using an automated ChatGPT-based classifier), and diverse (spanning topics from science and technology to entertainment and geography)[^2].

The benchmark addresses a core problem in modern AI: language models frequently generate confident but factually incorrect responses. By focusing exclusively on short-form factual queries with clear ground-truth answers, SimpleQA provides a clean, reproducible signal for measuring progress on factuality. At the time of its release, no frontier model achieved more than 50% [accuracy](/wiki/accuracy) on the benchmark, with OpenAI's o1-preview leading at 42.7%[^2]. Subsequent OpenAI models pushed parametric (closed-book) scores higher: GPT-4.5 reached 62.5% in February 2025[^3], and the August 2025 GPT-5 system card reported gpt-5-thinking at 55% accuracy with a 40% hallucination rate on SimpleQA[^12]. By late 2025, attention had shifted toward the curated [SimpleQA Verified](/wiki/simpleqa) subset (released September 2025) as researchers found that the very high scores some models posted on the original benchmark could not always be reproduced under stricter conditions[^7].

## Background and Motivation

### What problem does SimpleQA address?

One of the most persistent challenges in deploying large language models is their tendency to produce false or unsubstantiated outputs, a phenomenon known as [hallucination](/wiki/hallucination). Language models can state incorrect facts with high confidence, making it difficult for users to distinguish reliable answers from fabricated ones. This problem is especially concerning in high-stakes applications like healthcare, legal research, and education, where factual accuracy is essential[^4].

Prior to SimpleQA, several benchmarks existed for evaluating model truthfulness and factuality, including TruthfulQA and [MMLU](/wiki/mmlu). However, these benchmarks either conflated factuality with reasoning ability, relied on subjective judgments, or had become saturated as models improved. OpenAI identified the need for a benchmark that isolated factual recall from other cognitive tasks, focused on questions with unambiguous answers, and remained challenging for frontier models[^2].

### What were the design goals?

The SimpleQA authors articulated three core design properties that guided the benchmark's construction[^2]:

1. **Challenging**: Questions were adversarially collected against GPT-4o and GPT-3.5 responses. During the data collection phase, each question was required to cause at least one frontier model to hallucinate, ensuring the benchmark would differentiate among top-performing systems.

2. **Grading simplicity**: Every question has a single, indisputable correct answer. This removes the ambiguity that plagues open-ended evaluation and allows automated grading with high reliability.

3. **Diversity**: The question set covers a broad range of topics, answer types, and source documents, reducing the risk that a model could perform well simply by memorizing a narrow domain.

## Dataset Construction

### How was the dataset built?

SimpleQA was built through a careful two-stage process involving human annotators (referred to as "AI trainers" in the paper)[^2]:

**Stage 1: Question and Answer Creation**

In the first stage, AI trainers browsed the web and created short, fact-seeking questions along with their reference answers. Each question had to satisfy the following criteria:

- The question must have a single, indisputable answer.
- The answer must not change over time (time-invariant facts only).
- The question should be specific enough that the intended answer is unambiguous (for example, "Which city..." rather than "Where...").
- The trainer must provide a supporting web link for the reference answer.

Additionally, trainers reviewed four OpenAI model responses (from GPT-4o and GPT-3.5) and only continued with questions where at least one model produced an incorrect answer. This adversarial filtering step ensured the benchmark would remain challenging for frontier models.

**Stage 2: Independent Verification**

A second, independent AI trainer answered each question without seeing the original answer. A ChatGPT classifier was also used to detect potential violations of the question criteria (such as ambiguity or time-dependent answers). Only questions where both trainers' answers agreed were retained in the final dataset. Grammar improvements were applied without altering the factual content.

**Quality Validation**

As a final check, a third AI trainer independently answered a random sample of 1,000 questions from the dataset. This validation step revealed an approximate 3% error rate in the benchmark itself, meaning roughly 97% of questions have verified correct ground-truth answers[^2].

Of the 56 cases (5.6% of the 1,000-question sample) where the third trainer's answer was initially graded as incorrect, manual review identified 15 false negatives from the automated grader. Seven errors involved incomplete but partially correct answers, and six involved misreadings by the trainer. The remaining discrepancies (roughly 2.8%) stemmed from genuinely ambiguous questions, contradictory reputable sources, or questions that had multiple valid answers[^2].

### What topics and domains does SimpleQA cover?

The 4,326 questions span a wide range of knowledge domains, classified using ChatGPT:

| Domain | Number of Questions | Percentage |
| --- | --- | --- |
| Science & Technology | 858 | 19.8% |
| Politics | 709 | 16.4% |
| Art | 550 | 12.7% |
| History | ~475 | ~11.0% |
| Entertainment | ~430 | ~10.0% |
| Geography | ~390 | ~9.0% |
| Other (sports, business, general knowledge) | ~914 | ~21.1% |

### Answer Type Distribution

The benchmark captures a variety of factual answer types:

| Answer Type | Percentage | Example Question |
| --- | --- | --- |
| Dates | 32.8% | "What day, month, and year was Carrie Underwood's album 'Cry Pretty' certified Gold by the RIAA?" |
| Person names | 24.1% | "Who received the IEEE Frank Rosenblatt Award in 2010?" |
| Numbers | 15.3% | "How many episodes are in the first season of Bridgerton?" |
| Places | 9.9% | "On which U.S. TV station did the Canadian reality series To Serve and Protect debut?" |
| Other | 18.0% | Various factual responses (titles, organizations, objects) |

### Source Distribution

AI trainers were required to provide a web link supporting each reference answer. The distribution of source domains shows a heavy reliance on established encyclopedias and reference sites[^2]:

| Source Domain | Approximate Question Count |
| --- | --- |
| Wikipedia | ~3,500 |
| Fandom.com | ~410 |
| Academic domains | ~154 |
| IMDb | ~121 |
| Other | ~141 |

The strong representation of Wikipedia reflects its role as the most comprehensive and accessible general-purpose reference, though the inclusion of Fandom, IMDb, and academic sources ensures coverage of entertainment, pop culture, and specialized knowledge domains.

## Evaluation Methodology

### How is SimpleQA graded?

SimpleQA uses a three-category grading scheme that distinguishes it from binary correct/incorrect benchmarks[^2]:

| Grade | Definition | Example |
| --- | --- | --- |
| **Correct** | The model's answer fully contains the reference answer without any contradictions | Q: "Capital of France?" A: "The capital of France is Paris." |
| **Incorrect** | The model's answer contradicts the reference answer in any way | Q: "Capital of France?" A: "The capital of France is London." |
| **Not Attempted** | The model's response does not provide the requested information and does not contain contradictions | Q: "Capital of France?" A: "I'm not sure about the answer to that question." |

The "not attempted" category is a critical innovation. It allows the benchmark to measure not just whether a model gets answers right, but whether a model knows what it does not know. A well-calibrated model should attempt questions it is likely to answer correctly and decline questions where it is uncertain, rather than guessing and producing a hallucination.

### How does automated grading with ChatGPT work?

Rather than relying on human graders for the full 4,326-question set, SimpleQA uses a prompted ChatGPT classifier to automate grading[^2]. The classifier receives both the model's predicted answer and the ground-truth reference answer, then outputs one of three labels: CORRECT, INCORRECT, or NOT_ATTEMPTED.

The grading prompt (provided in Appendix A of the paper) includes detailed instructions and worked examples for each category. To validate the classifier's reliability, the authors manually reviewed 100 examples from each grade category. Out of 300 total reviewed examples, only two disagreements were found between the automated grader and human judgment, confirming the high reliability of the automated approach[^2].

This automated grading pipeline is a practical advantage of SimpleQA. Because the questions have unambiguous answers and the grading criteria are well defined, the benchmark can be run at scale without human involvement in the evaluation loop.

### What metrics does SimpleQA report?

SimpleQA reports several complementary metrics that together provide a comprehensive view of model factuality[^2]:

| Metric | Formula | Description |
| --- | --- | --- |
| **Correct (overall)** | Correct / Total | The percentage of all questions the model answered correctly. This is the primary accuracy measure. |
| **Correct Given Attempted** | Correct / (Correct + Incorrect) | The accuracy rate among questions the model actually tried to answer, excluding those it declined. Analogous to precision. |
| **Not Attempted Rate** | Not Attempted / Total | The percentage of questions the model chose not to answer. This measures how often the model exercises restraint. |
| **F-score** | Harmonic mean of Correct and Correct Given Attempted | A single-number summary that balances raw accuracy with precision on attempted questions. |

The F-score is particularly useful because it penalizes models that achieve high "Correct Given Attempted" scores by only answering a small number of easy questions while declining most of the benchmark. Conversely, it penalizes models that attempt everything but get many answers wrong.

### Parametric (Closed-Book) Evaluation

SimpleQA is intended as a measurement of *parametric* knowledge: facts encoded in the model's weights rather than retrieved at inference time. Standard evaluations therefore run the model without web search, retrieval-augmented generation, or external tool calls. This distinction has become important as several leaderboards now report SimpleQA-style numbers for systems that include retrieval, producing accuracies above 90% that do not reflect the same closed-book capability the original paper measured[^6][^13]. Anthropic's [Claude](/wiki/claude) Opus 4.6 system card, for example, includes a "no-tools" SimpleQA result alongside other factuality measurements precisely to preserve this distinction[^14].

## Model Performance

### How do frontier models score on SimpleQA?

The initial SimpleQA paper reported results for eight models from OpenAI and Anthropic[^2]:

| Model | Correct | Not Attempted | Incorrect | Correct Given Attempted | F-score |
| --- | --- | --- | --- | --- | --- |
| OpenAI o1-preview | 42.7% | 9.2% | 48.1% | 47.0% | 44.8% |
| [GPT-4](/wiki/gpt-4)o | 38.2% | 1.0% | 60.8% | 38.0% | 38.4% |
| [Claude](/wiki/claude) 3.5 Sonnet | 28.9% | 35.0% | 36.1% | 44.5% | 35.0% |
| GPT-4 Turbo | 24.2% | N/A | N/A | N/A | N/A |
| Claude 3 Opus | 23.5% | 39.6% | 36.9% | 38.8% | 29.3% |
| OpenAI o1-mini | 8.1% | 28.5% | 63.4% | 11.3% | 9.4% |
| GPT-4o-mini | 8.6% | 0.9% | 90.5% | 8.7% | 8.6% |
| Claude 3 Sonnet | 5.7% | 75.0% | 19.3% | 22.9% | 9.2% |
| Claude 3 Haiku | 5.1% | 75.3% | 19.6% | 20.6% | 8.2% |

Several patterns emerged from these results:

- **No model exceeded 50% accuracy**, confirming the benchmark's difficulty.
- **Larger models consistently outperformed smaller variants** within the same family (GPT-4o vs. GPT-4o-mini, Claude 3 Opus vs. Claude 3 Haiku).
- **Claude models attempted far fewer questions** than GPT models. Claude 3.5 Sonnet left 35% of questions unanswered, while GPT-4o left only 1%. This reflects fundamentally different approaches to uncertainty: Claude models were more conservative, declining to answer when unsure.
- **The o1 reasoning models showed improved factuality.** The o1-preview model achieved the highest overall correct rate (42.7%) and the highest F-score (44.8%), suggesting that extended reasoning (chain-of-thought) at inference time helps models produce more accurate factual answers.

### OpenAI simple-evals Scores (2025)

As newer models were released, additional SimpleQA scores became available through OpenAI's simple-evals repository[^3][^5]:

| Model | SimpleQA Score (Correct %) |
| --- | --- |
| GPT-4.5 | 62.5% |
| o3 | 49.4% |
| o3-high | 48.6% |
| o1 | 42.6% |
| o1-preview | 42.4% |
| GPT-4.1 | 41.6% |
| GPT-4o (2024-08-06) | 40.1% |
| GPT-4o (2024-05-13) | 39.0% |
| GPT-4o (2024-11-20) | 38.8% |
| GPT-4 Turbo | 24.2% |
| o4-mini | 20.2% |
| o4-mini-high | 19.3% |
| GPT-4.1-mini | 16.8% |
| o3-mini-high | 13.8% |
| o3-mini | 13.4% |
| o3-mini-low | 13.0% |
| GPT-4o-mini | 9.5% |
| o1-mini | 7.6% |
| GPT-4.1-nano | 7.6% |

GPT-4.5 (released February 2025) became the first OpenAI model to cross the 50% threshold on the original SimpleQA, scoring 62.5%[^3]. OpenAI attributed this improvement to the model's greater world knowledge and reduced tendency to hallucinate.

### GPT-5 System Card Results (August 2025)

The official GPT-5 system card, published on August 13, 2025, reported SimpleQA accuracy and hallucination rates for the GPT-5 family alongside several earlier OpenAI models[^12]:

| Model | SimpleQA Accuracy | Hallucination Rate |
| --- | --- | --- |
| gpt-5-thinking | 55% | 40% |
| OpenAI o3 | 54% | 46% |
| gpt-5-main | 46% | 47% |
| GPT-4o | 44% | 52% |
| OpenAI o4-mini | 24% | 75% |
| gpt-5-thinking-mini | 22% | 26% |
| gpt-5-thinking-nano | 11% | 31% |

The system card noted that gpt-5-thinking showed a slight improvement in hallucination rate over o3, and that thinking-mini outperformed o4-mini on both metrics. Hallucination rate here is the fraction of attempted answers that were incorrect, complementing the accuracy figure[^12].

### Why do some leaderboards report scores above 90%?

Public leaderboards that track SimpleQA performance across many providers have reported scores well above 90% for several 2025-2026 models, including DeepSeek-V3.2-Exp (97.1%), Grok 4 Fast (95.0%), and DeepSeek-V3.1 (93.4%)[^6]. These numbers are difficult to reconcile with the GPT-5 system card's 55% closed-book result and most likely reflect either web search / retrieval-augmented configurations or training contamination, since the benchmark questions and reference answers are public. The original SimpleQA paper explicitly defines the task as a parametric-knowledge evaluation, and OpenAI's reference implementation does not provide tools to the model[^2][^3]. Headline scores above 90% should therefore be interpreted as upper bounds for an entire system (model plus tools) rather than as gains in the model's intrinsic factual knowledge, and the September 2025 release of [SimpleQA Verified](/wiki/simpleqa) was motivated in part by these difficulties[^7].

### SimpleQA Verified Comparison

When OpenAI's and other vendors' models are re-evaluated on the 1,000-question SimpleQA Verified subset (see below), parametric scores remain in the same range as on the original. Google reported the following F1-scores in the September 2025 launch[^7]:

| Model | SimpleQA Verified F1 | Change vs. original SimpleQA |
| --- | --- | --- |
| Gemini 2.5 Pro | 55.6% | +0.5 |
| GPT-5 | 52.3% | +1.8 |
| o3 | 51.9% | +1.9 |
| GPT-4.1 | 39.9% | -1.0 |
| GPT-4o | 34.9% | -3.5 |
| DeepSeek R1 | 33.3% | +1.4 |
| Claude Opus 4 | 28.3% | -4.0 |
| Gemini 2.5 Flash | 28.2% | -1.4 |
| GPT-5 mini | 24.6% | +1.1 |
| o4-mini | 23.4% | +2.9 |

Following the launch of Gemini 3 Pro on November 18, 2025, Google reported a state-of-the-art SimpleQA Verified score of 72.1% for the new model, a substantial jump over Gemini 2.5 Pro's 54.5% and an approximate 40-percentage-point gap above the next-best contemporaneous competitor on this evaluation[^15][^16].

## Calibration Analysis

### What does SimpleQA reveal about model calibration?

One of SimpleQA's most important contributions is its measurement of model calibration: does a model's expressed confidence align with its actual accuracy? A perfectly calibrated model would be correct exactly X% of the time on questions where it states X% confidence[^2]. As OpenAI framed it, the benchmark is "a simple, targeted evaluation for whether models know what they know."[^2]

### Stated Confidence Method

The first calibration approach asks models to explicitly state their confidence as a percentage (0-100%) alongside each answer. Researchers then group answers by stated confidence level and measure the actual accuracy within each group.

Results from the paper showed a positive correlation between stated confidence and accuracy across all tested models. However, models consistently overstated their confidence. For instance, when models claimed 90% confidence, their actual accuracy was often substantially lower. This overconfidence is a hallmark of the hallucination problem: models are not just wrong, they are wrong while being confident they are right[^2].

### Response Frequency Method

The second calibration approach is more indirect. The same question is posed to the model 100 times at temperature 1 (the sampling temperature that introduces randomness into responses). String matching groups the different answers together, and only the most frequent answer for each question is considered.

The intuition behind this method is that if a model repeatedly produces the same answer across many samples, it has a strong internal representation of that fact. If it produces different answers each time, the model is uncertain.

Results showed that accuracy increases with answer frequency across all models. The o1-preview model demonstrated the strongest calibration using this method: the frequency of a given response was roughly equivalent to the accuracy of that response. Larger models were more calibrated than smaller ones in general[^2].

### Implications for Trustworthy AI

The calibration findings have direct implications for deploying language models in real-world applications. Models that are well-calibrated can be more safely used in systems where they are allowed to abstain rather than guess. The "not attempted" mechanism in SimpleQA directly rewards this behavior, incentivizing model developers to build systems that express appropriate uncertainty.

## Limitations and Criticisms

### What are the limitations of SimpleQA?

The SimpleQA paper acknowledges several limitations[^2]:

- **English only**: The benchmark covers only English-language questions, limiting its applicability to evaluating models in other languages.
- **Short-form factuality only**: SimpleQA measures factual recall under a constrained setting of brief queries with single correct answers. Whether improvements on short-form factuality transfer to longer, multi-claim responses remains an open research question.
- **Static dataset**: The fixed set of 4,326 questions creates a risk of overfitting as models are repeatedly evaluated on the same questions.
- **Temporal cutoff**: All facts are verified as of December 31, 2023. Questions about events after this date are excluded, and the benchmark will need periodic updates.

### Criticisms from the Research Community

Following its release, researchers identified additional concerns with the original SimpleQA dataset[^7]:

- **Noisy and incorrect labels**: Despite the two-stage verification process, some ground-truth answers in the dataset are incorrect or ambiguous.
- **Topical biases**: The dataset disproportionately features certain topics and question formats. For example, 119 questions (2.7% of the dataset) ask about Colombian municipality founding dates, reflecting individual annotator preferences rather than a balanced knowledge distribution.
- **Question redundancy**: Many questions are semantically similar or have significant lexical overlap, meaning a model could achieve inflated scores by learning patterns rather than genuinely knowing facts.
- **Overrepresentation of certain answer types**: Dates account for 32.8% and person names for 24.1% of all answers, creating an uneven evaluation of factual knowledge breadth.
- **Data contamination risk**: Because the benchmark's questions are publicly available, there is a risk that model training data could include the questions or closely related content, inflating scores without reflecting genuine factual recall.

These issues create what the SimpleQA Verified authors describe as a "noisy evaluation signal," making it difficult to determine whether performance gains stem from genuine improvements in factual recall or from models overfitting to the benchmark's specific quirks[^7].

### Deprecation of OpenAI's Public Leaderboard

In July 2025, OpenAI announced that the simple-evals repository would no longer be updated with new model scores for SimpleQA, HealthBench, or BrowseComp, although reference implementations would remain available[^5]. The decision effectively shifted the role of maintaining a vendor-neutral SimpleQA leaderboard to community trackers and to the SimpleQA Verified effort.

## SimpleQA Verified

### What is SimpleQA Verified?

In September 2025, researchers from Google (Lukas Haas, Gal Yona, Giovanni D'Antonio, Sasha Goldshtein, and Dipanjan Das) released SimpleQA Verified, a curated subset of 1,000 questions derived from the original SimpleQA benchmark[^7]. The goal was to provide a cleaner, more reliable evaluation instrument that addressed the known limitations of the original dataset, and it is explicitly designed to be "evaluated without any tools (i.e. search)" so that it isolates parametric factual knowledge[^7]. The dataset, evaluation code, and leaderboard are hosted on Kaggle[^7].

### How was SimpleQA Verified curated?

SimpleQA Verified was created through a rigorous multi-stage filtering process that removed 76.9% of the original questions:

| Filtering Step | Questions Removed | Purpose |
| --- | --- | --- |
| Duplicate source documents | 28.5% | Reduce annotator bias from repeated sources |
| Semantic de-duplication (Gemini embeddings, 0.77 threshold) | 7.2% | Remove semantically similar questions |
| TF-IDF de-duplication (0.4 threshold) | 7.2% | Remove lexically overlapping questions |
| Publisher robots.txt compliance | 30.4% | Respect web publisher crawling preferences |
| Answer type and topic rebalancing | 34.3% | Ensure diverse coverage across knowledge domains |
| Conflicting source reconciliation (non-numeric) | 8.3% | Verify ground-truth accuracy |
| Conflicting source reconciliation (numeric) | 3.9% | Verify numerical answer accuracy |
| Difficulty-based selection | 6.8% | Maintain benchmark challenge level |

The resulting 1,000-question set features more balanced topic coverage, verified ground-truth answers, and reduced redundancy compared to the original.

### Updated Grading

SimpleQA Verified also modified the autorater prompt, with changes focused on forcing direct answers, preventing credit for lucky guesses embedded in lengthy responses, and improving the grading of numeric answer types[^7].

### Performance Comparison

On SimpleQA Verified at launch (September 2025), Gemini 2.5 Pro held the top F1 score at 55.6%, followed by GPT-5 at 52.3% and o3 at 51.9%[^7]. Two months later, [Gemini](/wiki/gemini) 3 Pro reached 72.1%, more than 16 points above any previously published result[^15][^16].

## Related Benchmarks and Variants

### How does SimpleQA compare with other factuality benchmarks?

| [Benchmark](/wiki/benchmark) | Focus | Questions | Grading | Key Difference from SimpleQA |
| --- | --- | --- | --- | --- |
| **SimpleQA** | Short-form factuality | 4,326 | Automated (3-way) | Adversarially collected, single-answer |
| **SimpleQA Verified** | Short-form factuality (refined) | 1,000 | Automated (improved) | Cleaned version with bias reduction |
| **TruthfulQA** | Truthfulness and common misconceptions | 817 | Human + automated | Tests resistance to common falsehoods |
| [MMLU](/wiki/mmlu) | Comprehensive knowledge and reasoning | 14,042 | Multiple choice | Broader scope, includes reasoning |
| **TriviaQA** | Trivia knowledge | 95,000+ | Exact match | Larger but less curated |
| **GPQA** | Graduate-level expert knowledge | 448 | Multiple choice | Domain-expert difficulty |

### Multilingual Variants

**Chinese SimpleQA** was introduced in November 2024 as the first comprehensive Chinese-language factuality benchmark following the SimpleQA methodology[^8]. Published at ACL 2025, it contains 3,000 high-quality questions spanning six major topics with 99 diverse subtopics. The benchmark shares SimpleQA's core properties (diverse, high-quality, static, easy-to-evaluate) but is tailored to Chinese language and culture. Results showed that DeepSeek-V3 performed particularly well on Chinese SimpleQA, outperforming GPT-4o and Claude models on Chinese-language factual questions.

### Multimodal Extensions

The SimpleQA framework has been extended beyond text:

- **SimpleVQA** (2025): The first multimodal factuality benchmark, extending SimpleQA's approach to visual question answering. It covers nine different visual QA tasks across nine topics, evaluating whether multimodal large language models can answer factual questions about images[^9].

- **VisualSimpleQA** (2025): A related benchmark that decouples vision and knowledge capabilities in large vision-language models for fact-seeking question answering, with well-defined difficulty criteria guiding the annotation process[^10].

- **Video SimpleQA** (2025): The first comprehensive benchmark tailored for factuality evaluation in video contexts, extending the SimpleQA methodology to questions about video content[^11].

## Technical Implementation

### Is SimpleQA open source and how do you run it?

SimpleQA's evaluation code is open-sourced as part of OpenAI's simple-evals repository on GitHub under an MIT license. The implementation is lightweight by design, consisting of a Python script that:

1. Loads the 4,326 questions from the dataset.
2. Queries the target model with each question.
3. Sends the model's response along with the reference answer to the ChatGPT grading classifier.
4. Aggregates the CORRECT, INCORRECT, and NOT_ATTEMPTED grades into the reported metrics.

The dataset itself is available on Hugging Face, and the grading prompt is published in the paper's appendix, allowing full reproducibility[^2].

As of July 2025, OpenAI announced that the simple-evals repository would no longer be updated with new model scores, though it would continue to host reference implementations for SimpleQA, HealthBench, and BrowseComp[^5].

### Example Questions

The following examples from the paper illustrate the range and difficulty of SimpleQA questions[^2]:

| Question | Reference Answer | Domain |
| --- | --- | --- |
| Who received the IEEE Frank Rosenblatt Award in 2010? | Michio Sugeno | Science & Technology |
| On which U.S. TV station did the Canadian reality series *To Serve and Protect* debut? | KVOS-TV | Entertainment |
| What day, month, and year was Carrie Underwood's album 'Cry Pretty' certified Gold by the RIAA? | October 23, 2018 | Art / Music |
| What is the first and last name of the woman whom British linguist Bernard Comrie married in 1985? | Akiko Kumahira | History / People |

These questions demonstrate SimpleQA's emphasis on specific, verifiable facts that require precise knowledge rather than general reasoning.

## Research Impact and Applications

### Contributions to AI Safety

SimpleQA has become a standard reference point in discussions of AI safety and reliability. Its contributions include:

- **Standardized factuality metric**: Before SimpleQA, there was no widely adopted benchmark focused purely on short-form factual accuracy with automated grading. SimpleQA filled this gap and has been cited extensively in model release announcements and technical reports, including the GPT-5 system card[^12] and the Gemini 3 launch[^15].
- **Quantified hallucination rates**: The benchmark provided concrete numbers showing that even the best parametric-only models hallucinate on roughly 40-50% of factual questions, giving the research community a clear target for improvement[^12].
- **Calibration framework**: The dual calibration methods (stated confidence and response frequency) introduced a practical framework for assessing whether models understand their own knowledge limitations.
- **Influence on model development**: Multiple major model releases in 2025-2026 cited SimpleQA or SimpleQA Verified performance as evidence of improved factuality, including GPT-4.5, GPT-5, Gemini 2.5 Pro, Gemini 3 Pro, and DeepSeek-V3[^3][^12][^15].

### Practical Applications

- **Model development and selection**: Organizations evaluating language models for deployment use SimpleQA as a factuality screening tool, alongside other benchmarks.
- **Pre-deployment safety testing**: SimpleQA scores help identify models prone to hallucination before they are deployed in production systems.
- **Hallucination research**: Researchers studying the mechanisms behind hallucination use SimpleQA to measure the effectiveness of mitigation techniques such as retrieval augmentation, confidence-based abstention, and improved training data curation.
- **Product comparison**: Commercial AI providers reference SimpleQA scores when comparing the factual reliability of competing models.

## Future Directions

Several areas of ongoing and future work build on the SimpleQA framework:

1. **Multilingual expansion**: While Chinese SimpleQA exists, the framework has yet to be extended to most of the world's languages. Future work is expected to cover Japanese, Korean, Arabic, and other major languages.
2. **Dynamic evaluation**: Static benchmarks risk contamination and overfitting over time. Researchers are exploring methods for generating new SimpleQA-style questions continuously.
3. **Long-form factuality**: SimpleQA deliberately restricts itself to short-form answers. Extending the methodology to evaluate factual accuracy in longer, multi-paragraph responses is an active research area.
4. **Domain-specific benchmarks**: Specialized versions of SimpleQA for medical, legal, and scientific knowledge could provide more targeted evaluation for high-stakes applications.
5. **Real-time fact-checking**: The SimpleQA grading framework could be adapted for real-time monitoring of language model outputs in production, flagging potential hallucinations as they occur.
6. **Separating parametric and retrieval factuality**: As tool-augmented scores diverge sharply from closed-book scores, evaluation suites are increasingly reporting both, with SimpleQA Verified positioned as the primary parametric benchmark[^7][^12].

## See Also

- [Hallucination](/wiki/hallucination)
- [Benchmark](/wiki/benchmark)
- [MMLU](/wiki/mmlu)
- [OpenAI](/wiki/openai)
- [Accuracy](/wiki/accuracy)
- [LLM](/wiki/llm)
- [Gemini](/wiki/gemini)
- [Claude](/wiki/claude)
- TruthfulQA
- Question Answering

## References

[^1]: OpenAI. "Introducing SimpleQA." OpenAI Blog, October 30, 2024. https://openai.com/index/introducing-simpleqa/

[^2]: Wei, Jason, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. "Measuring short-form factuality in large language models." arXiv preprint arXiv:2411.04368, November 7, 2024. https://arxiv.org/abs/2411.04368

[^3]: OpenAI. "simple-evals: Lightweight library for evaluating language models." GitHub, 2024-2025. https://github.com/openai/simple-evals

[^4]: Ji, Ziwei, et al. "Survey of Hallucination in Natural Language Generation." ACM Computing Surveys, 2023.

[^5]: OpenAI. "simple-evals README." GitHub, July 2025. https://github.com/openai/simple-evals/blob/main/README.md

[^6]: LLM Stats. "SimpleQA Benchmark Leaderboard." Accessed May 2026. https://llm-stats.com/benchmarks/simpleqa

[^7]: Haas, Lukas, Gal Yona, Giovanni D'Antonio, Sasha Goldshtein, and Dipanjan Das. "SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge." arXiv preprint arXiv:2509.07968, September 2025. https://arxiv.org/abs/2509.07968

[^8]: He, Yuliang, et al. "Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025. https://arxiv.org/abs/2411.07140

[^9]: Cheng, et al. "SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models." ICCV 2025. https://arxiv.org/abs/2502.13059

[^10]: "VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering." arXiv preprint arXiv:2503.06492, 2025. https://arxiv.org/abs/2503.06492

[^11]: "Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models." arXiv preprint arXiv:2503.18923, 2025. https://arxiv.org/abs/2503.18923

[^12]: OpenAI. "GPT-5 System Card." August 13, 2025. https://cdn.openai.com/gpt-5-system-card.pdf (also published as arXiv:2601.03267).

[^13]: Price Per Token. "SimpleQA Leaderboard 2026 - Compare AI Model Scores." Accessed May 2026. https://pricepertoken.com/leaderboards/benchmark/simpleqa

[^14]: Anthropic. "Claude Opus 4.6 System Card." February 2026. https://www-cdn.anthropic.com/0dd865075ad3132672ee0ab40b05a53f14cf5288.pdf

[^15]: Google. "Gemini 3: Introducing the latest Gemini AI model from Google." Google Blog, November 18, 2025. https://blog.google/products-and-platforms/products/gemini/gemini-3/

[^16]: Epoch AI. "SimpleQA Verified." Accessed May 2026. https://epoch.ai/benchmarks/simple-qa-verified