# MMLU

> Source: https://aiwiki.ai/wiki/mmlu
> Updated: 2026-07-29
> Fact-checked: 2026-07-29
> Categories: AI Benchmarks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "MMLU." aiwiki.ai, 29 Jul 2026. https://aiwiki.ai/wiki/mmlu
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

MMLU, short for Massive Multitask Language Understanding, is an English-language [AI benchmark](https://aiwiki.ai/wiki/benchmark) for evaluating broad academic and professional knowledge through four-option multiple-choice questions. [Dan Hendrycks](https://aiwiki.ai/wiki/dan_hendrycks), Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and [Jacob Steinhardt](https://aiwiki.ai/wiki/jacob_steinhardt) introduced it in a paper first submitted in September 2020 and published at the 2021 [International Conference on Learning Representations](https://aiwiki.ai/wiki/iclr). The benchmark contains 57 subjects spanning the humanities, social sciences, science and mathematics, business, law, medicine, and other fields.[1]

MMLU became a common test for comparing [large language models](https://aiwiki.ai/wiki/large_language_model), but an MMLU percentage is not self-interpreting. The original evaluation used up to five fixed demonstrations from the same subject and selected among the answer-label tokens A, B, C, and D by their probabilities. Other evaluators generate an answer label, change the prompt, reorder choices, use a different dataset snapshot, or aggregate subjects differently. Those choices can produce meaningfully different results for the same model.[2][4][5][8]

Later research has also identified flaws that constrain what a score can establish. MMLU-Redux estimated that 6.49 percent of the test questions contain an error, based on expert review of 5,700 questions.[6] Separate work found evidence consistent with possible test-set exposure in some models, while also emphasizing that its diagnostic did not directly prove which training examples a model had seen.[7] MMLU remains useful when the dataset revision, prompt, scoring rule, aggregation method, and contamination controls are reported, but scores produced under different protocols should not be treated as directly interchangeable.

## Purpose and design

MMLU was designed to test knowledge and problem solving across many subjects without supplying a large task-specific training set. This differs from a conventional supervised benchmark in which a model learns each task from many labeled training examples. The original authors instead evaluated zero-shot, few-shot, and transfer settings to ask how much relevant knowledge a model had acquired before the test.[1]

Every scored item presents one question and four answer options. A correct response is one of four labels, so uniform random guessing has an expected accuracy of 25 percent. The simple output space makes automatic grading straightforward. It does not make all questions easy: the source material includes professional examinations, university courses, high-school subjects, and specialized textbook questions.[1]

The 57 subjects are grouped into four broad areas in the paper:

| Broad area | Examples from the original subject list |
|---|---|
| Humanities | Formal logic, international law, jurisprudence, moral disputes, philosophy, prehistory, world religions |
| Social sciences | Econometrics, high-school geography, macroeconomics, microeconomics, psychology, sociology |
| STEM | Abstract algebra, anatomy, astronomy, computer science, engineering, mathematics, physics, statistics |
| Other | Accounting, business ethics, global facts, management, marketing, medicine, nutrition |

These broad areas are reporting groups, not equally sized or internally uniform constructs. A subject can mix recall, calculation, interpretation, and domain-specific reasoning. The original paper used the spread of subject results to show that one overall number can conceal major strengths and weaknesses.[1]

### Question provenance

Graduate and undergraduate students manually collected the questions from sources that the paper described as freely available online. Examples included practice material for the Graduate Record Examination and United States Medical Licensing Examination, questions designed for undergraduate courses, and reader questions associated with Oxford University Press books. The collection therefore repackages questions from multiple pre-existing sources rather than creating every item from a single specification.[1]

The official archive includes a README directing evaluators to answer questions according to human knowledge as of January 1, 2020. This temporal scope matters for items whose answers could change. MMLU should not be used as a general test of knowledge about later events without separately updating and validating the questions.[3]

The official repository includes an MIT License file.[2] That repository-level license record does not document the provenance of every underlying question beyond the source description in the paper. Researchers who redistribute or adapt the corpus should keep the distinction between the repository and the third-party material from which questions were collected.

## Dataset splits and count discrepancy

The paper describes a development set for few-shot examples, a validation set for hyperparameter selection, and a held-out test set. It reports 15,908 questions in total, including 285 development questions, 1,540 validation questions, and 14,079 test questions.[1] The three printed split counts add to 15,904 rather than 15,908.

A row-by-row audit of the official Berkeley archive, performed on July 28, 2026 (UTC), produced another set of counts:[3]

| Split | Paper-reported rows | Rows in the official archive audited July 28, 2026 (UTC) |
|---|---:|---:|
| Development | 285 | 285 |
| Validation | 1,540 | 1,531 |
| Test | 14,079 | 14,042 |
| Development, validation, and test combined | 15,904 by addition, although the paper states 15,908 total | 15,858 |
| Auxiliary training data | Not included in the paper's three scored split counts | 99,842 |

The archive has 57 files in each of the development, validation, and test directories. Every development file contains five examples, matching the original five-shot design. The archive also has eight auxiliary-training files. Its README names MCTest, RACE, ARC, and OpenBookQA among their sources. Including those auxiliary rows produces 115,700 rows, but they are not part of the standard MMLU test score.[3]

The reviewed primary sources do not explain the four-question arithmetic inconsistency in the paper or the later difference between the paper and archive counts. Consequently, a reproducible result should identify the exact dataset artifact rather than cite only the benchmark name. The widely used 14,042-item test count refers to the current archive, whereas 14,079 is the count printed in the original paper.[1][3][8]

## Original evaluation protocol

### Five-shot prompting

For each subject, the original protocol formed a prompt with an instruction naming the subject, followed by as many as five fixed question-and-answer demonstrations from that subject. It then appended the test question and ended with `Answer:`. The [GPT-3](https://aiwiki.ai/wiki/gpt-3) evaluator compared the probabilities of the answer-label tokens and selected the most probable label. If the prompt exceeded the model's context limit, the reference implementation removed demonstrations until it fit.[1][2]

In this context, five-shot refers to five in-prompt examples, not five gradient updates or five training records sampled anew for every test question. The development split fixes one set of five demonstrations for each subject. Zero-shot evaluation omits those demonstrations. The two settings therefore measure related but different conditions.[1]

The exact answer-scoring operation also matters. The reference code queries log probabilities for labels represented with a preceding space, such as ` A`, then chooses the largest value.[2] A system that instead generates unrestricted text and extracts a letter can fail through formatting or answer-extraction behavior even when its underlying preference among the four choices is unchanged.

### Overall accuracy

The original paper describes the metric as classification accuracy across all examples and tasks. Its reference implementation concatenates the per-question correctness values from all subjects and takes one mean.[1][2] The overall score is therefore an item-weighted, or micro-averaged, accuracy:

`overall accuracy = number of correct test answers / number of evaluated test questions`

It is not an unweighted mean that gives every subject exactly one fifty-seventh of the total. Larger subjects contribute more questions and therefore more weight. The [LM Evaluation Harness](https://aiwiki.ai/wiki/lm_evaluation_harness) documentation likewise identifies its MMLU aggregate as a micro average weighted by subtask size.[5]

A macro-average over subject accuracies is a legitimate alternative statistic, but it is a different metric and should be labeled as such. The same applies to averages over the four broad areas, normalized scores, and subset-only results.

### Reimplementations

Stanford's [HELM](https://aiwiki.ai/wiki/helm) standardized MMLU evaluation uses all 57 subjects and the same five examples per subject, but grades a model's generated choice among A through D. Its researchers reported differences of as much as 5.0 percentage points between some creator-reported scores and their standardized measurements. They attributed the comparison problem to factors including unavailable prompts, private model snapshots, different inference interfaces, and more elaborate methods such as chain-of-thought prompting or uncertainty routing.[4]

These are not merely presentation details. A report saying that a model scored a particular percentage should specify at least:

- the dataset archive, revision, or checksum;
- all 57 subjects or the exact subset used;
- zero-shot, five-shot, or another number of demonstrations;
- the exact demonstrations and prompt template;
- answer order and answer-label symbols;
- token-probability, generated-label, cloze, hybrid, or another scoring method;
- item-weighted, subject-macro, or other aggregation;
- model identifier, model revision or access date, and inference settings;
- any decontamination rule, answer filtering, or post-processing.

Without these details, two values bearing the name MMLU may not measure the same procedure.

## Results reported in the original paper

The original 2021 paper reported the following overall accuracies. The rows do not all share one training regime: [RoBERTa](https://aiwiki.ai/wiki/roberta), [ALBERT](https://aiwiki.ai/wiki/albert), and GPT-2 were fine-tuned using other question-answering data and the MMLU development and validation sets; UnifiedQA was evaluated in a transfer setting without additional MMLU fine-tuning; GPT-3 used few-shot prompts.[1]

| Model or baseline | Original overall accuracy (%) |
|---|---:|
| Random baseline | 25.0 |
| RoBERTa | 27.9 |
| ALBERT | 27.1 |
| GPT-2 | 32.4 |
| UnifiedQA | 48.9 |
| GPT-3 Small, few-shot | 25.9 |
| GPT-3 Medium, few-shot | 24.9 |
| GPT-3 Large, few-shot | 26.0 |
| GPT-3 X-Large, few-shot | 43.9 |

For the 175-billion-parameter GPT-3 X-Large configuration, the paper also reported about 37.7 percent in the zero-shot setting. Its five-shot subject scores ranged from 26 percent on college chemistry to 69 percent on United States foreign policy. UnifiedQA's highest subject result was 82.5 percent on marketing. These results supported the authors' conclusion that performance was uneven across subjects, even when an aggregate result was above chance.[1]

The paper's human comparisons require a further qualification. Unspecialized Amazon Mechanical Turk participants achieved 34.5 percent. The stated 89.8 percent expert level was not measured by administering all 57 subjects to a single group of experts. It was an estimate assembled from approximately 95th-percentile performance on source examinations and the authors' educated guesses when corresponding figures were unavailable.[1] It should be described as an estimated reference point, not as a measured universal human ceiling.

The paper also examined [calibration](https://aiwiki.ai/wiki/calibration) for GPT-3. In its zero-shot experiments, average confidence and accuracy could differ by as much as 24 percentage points for a subject; five-shot results were better calibrated but still imperfect.[1] This finding applies to the evaluated GPT-3 configurations and protocol. It does not establish a fixed calibration error for later models.

## Reliability and validity

### Question and answer errors

MMLU-Redux is a later audit of MMLU, not a new release of the complete original dataset. Fourteen subject-matter annotators reviewed 100 questions from each of the 57 subjects, for 5,700 questions in total. They categorized unclear questions, unclear options, questions with no correct answer, questions with multiple correct answers, and incorrect reference labels. A stratified estimate placed the overall error rate at 6.49 percent.[6]

Error rates varied sharply by subject. The Redux study found an error in 57 percent of the 100 reviewed virology questions, including unclear questions and wrong, missing, or non-unique answers. This is a result for the reviewed sample, not proof that exactly 57 percent of every version of the full virology split is wrong. The study also found that excluding questions annotated as erroneous changed measured performance and altered model rankings for several analyzed subsets.[6]

The authors identified limits to their own audit. The remaining 8,342 test questions had not been manually annotated in that release, and the classification protocol could still reflect annotator bias. MMLU-Redux therefore provides evidence that source quality affects the benchmark, but it does not supply a fully corrected ground truth for every original item.[6]

### Possible training-data contamination

MMLU questions were drawn from material available on the public web, and the complete benchmark later became widely downloadable. This creates a [data contamination](https://aiwiki.ai/wiki/data_contamination) risk when a model's pretraining or post-training data are unknown.[1][7]

Deng and colleagues investigated this risk with a diagnostic called Testset Slot Guessing. It masks an incorrect option and asks a model to reconstruct the missing text after filtering highly correlated choices. On MMLU, ChatGPT and GPT-4 exactly reconstructed the masked option in 52 percent and 57 percent of evaluated cases, respectively. A deliberately contaminated ChatGPT model, fine-tuned on the MMLU test set, reached nearly 100 percent exact match, showing that the diagnostic is sensitive to known contamination.[7]

Those percentages are not measured contamination rates. The authors described the result as evidence of potential leakage, and noted that reconstruction relies on model instruction-following and inferred knowledge rather than direct access to the training corpus. Their separate retrieval method was limited to indexed portions of The Pile and C4 and used BM25. A high MMLU score or a successful reconstruction can raise concern, but neither alone proves that a particular test item occurred in training.[7]

### Answer-order and scoring sensitivity

Alzahrani and colleagues evaluated 11 models across more than 22 MMLU settings. Changes to answer order, answer-label symbols, and scoring method could substantially alter accuracies and rankings, with models moving by as many as eight rank positions across the studied multiple-choice settings.[8]

The study distinguished several mechanisms. Symbol scoring can reflect preferences for particular label tokens. Reordering the option text can expose position preferences. Cloze scoring avoids presenting the answer labels but changes the task and often lowers accuracy. The authors recommended a hybrid method as a practical way to reduce, rather than eliminate, some of these biases.[8]

Not every experiment covered the complete benchmark. A highlighted answer-shuffling analysis used three manually checked subjects: college chemistry, college mathematics, and global facts. The broader conclusions therefore combine full-benchmark and subset experiments. The same paper found that minor changes to instructions or few-shot examples generally affected rankings less than answer-choice and scoring changes in its evaluated models.[8]

### What MMLU does not measure

MMLU is a text-only, fixed-answer test. It does not directly test open-ended generation, factual citation, multimodal perception, tool use, interactive problem solving, long-running agents, or safety behavior. Its correct answers mostly reflect source material available by 2020. These are boundaries of the benchmark design, not defects that can be repaired by a higher score.[1][3]

Accuracy also combines different kinds of success. A correct label can come from subject knowledge, calculation, elimination, test-taking patterns, memorization, or chance. Because no explanation is required, ordinary MMLU scoring cannot by itself determine which mechanism produced a correct answer. Per-subject results, controlled perturbations, contamination checks, and complementary evaluations are needed to support narrower claims about capability.

## Later benchmarks and variants

Several later datasets include MMLU in their names but are separate benchmarks or audits. They should not be called versions of the original MMLU score without qualification.

| Name | Relationship to original MMLU |
|---|---|
| [MMLU-Redux](https://aiwiki.ai/wiki/mmlu_redux) | Expert audit and reannotation of 5,700 original test questions. The release records error types and, where applicable, suggested alternative answers; its reported re-evaluation filtered out erroneous items rather than replacing the original labels.[6] |
| [MMLU-Pro](https://aiwiki.ai/wiki/mmlu-pro) | A distinct 2024 benchmark with 12,032 questions in 14 domains, up to ten answer options, and a greater emphasis on multi-step reasoning. Its authors reported model drops of 16 to 33 percentage points relative to MMLU and lower prompt sensitivity in their tested settings.[9] |
| MMLU-CF | A 2025 benchmark designed to reduce contamination by collecting questions from a search domain of more than 200 billion webpages and applying three decontamination rules. It uses a public validation set and a closed test set; evaluations of more than 40 models produced score and ranking changes relative to original MMLU.[10] |
| [Global-MMLU](https://aiwiki.ai/wiki/global_mmlu) | A multilingual and culturally sensitive evaluation spanning 42 languages, with professional and community verification of translations.[11] |
| [MMMLU](https://aiwiki.ai/wiki/mmmlu) | OpenAI's professionally translated versions of the MMLU test split in 14 languages. It is a multilingual translation resource rather than the original English dataset.[12] |
| CMMLU | A separate Chinese multitask benchmark designed around Chinese language and cultural knowledge rather than a direct translation of every MMLU item.[13] |
| [MMLU-ProX](https://aiwiki.ai/wiki/mmlu_prox) | A multilingual extension of MMLU-Pro covering 29 languages, with 11,829 aligned questions per language and a 658-question Lite subset.[14] |

MMLU-Pro's ten-option, reasoning-focused design and use of [chain-of-thought](https://aiwiki.ai/wiki/chain_of_thought) evaluation make its percentages especially unsuitable for direct numerical comparison with original four-option MMLU results.[9] Likewise, multilingual variants introduce translation, cultural adaptation, and language-comparability questions that the original English benchmark was not designed to answer.[11][12][13][14]

## Interpreting a reported score

A defensible MMLU statement ties the number to its protocol. For example, "item-weighted accuracy on all 14,042 test rows from the audited official archive, using the original five fixed demonstrations and answer-token probability scoring" identifies a substantially clearer measurement than "MMLU five-shot."

For model comparisons, the strongest practice is to run every model through one evaluator and publish the prompts and per-item outputs. This reduces hidden variation, although it cannot eliminate differences in model access, tokenizer behavior, or training-data exposure. HELM follows this approach by exposing standardized scenarios, prompts, and predictions.[4] The LM Evaluation Harness similarly provides explicit task configurations and aggregation behavior.[5]

An overall result should be accompanied by per-subject scores when conclusions concern broad competence. Confidence intervals or paired item-level comparisons are preferable when differences are small. Researchers should also state whether known erroneous items were retained, excluded, or replaced, and whether the result uses original MMLU, MMLU-Redux labels, or a different derivative.

MMLU is best understood as one reproducible sample of multiple-choice performance, not a complete measure of [natural language understanding](https://aiwiki.ai/wiki/natural_language_understanding) or general intelligence. Its value lies in broad coverage and a simple metric. Its limits come from item quality, public availability, uneven subject sizes, protocol sensitivity, and the narrowness of fixed-choice evaluation.

## See also

- [Artificial Intelligence](https://aiwiki.ai/wiki/artificial_intelligence)
- [Few-Shot Learning](https://aiwiki.ai/wiki/few-shot_learning)
- [In-Context Learning](https://aiwiki.ai/wiki/in_context_learning)
- [T5](https://aiwiki.ai/wiki/t5)
- [Hugging Face](https://aiwiki.ai/wiki/hugging_face)
- [EleutherAI](https://aiwiki.ai/wiki/eleutherai)
- [Reasoning](https://aiwiki.ai/wiki/reasoning)

## References

1. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. "Measuring Massive Multitask Language Understanding." ICLR 2021. https://arxiv.org/abs/2009.03300
2. Hendrycks, D., et al. "Measuring Massive Multitask Language Understanding." Official GitHub repository. https://github.com/hendrycks/test
3. Hendrycks, D., et al. "MMLU official data archive." University of California, Berkeley. Accessed July 28, 2026 (UTC). https://people.eecs.berkeley.edu/~hendrycks/data.tar
4. Mai, Y., and Liang, P. "Massive Multitask Language Understanding (MMLU) on HELM." Stanford Center for Research on Foundation Models, May 1, 2024. https://crfm.stanford.edu/2024/05/01/helm-mmlu.html
5. EleutherAI. "Task Guide." Language Model Evaluation Harness documentation, "Group Configuration" section. https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md
6. Gema, A. P., et al. "Are We Done with MMLU?" Proceedings of NAACL 2025, pp. 5069-5096. https://aclanthology.org/2025.naacl-long.262/
7. Deng, C., Zhao, Y., Tang, X., Gerstein, M., and Cohan, A. "Investigating Data Contamination in Modern Benchmarks for Large Language Models." Proceedings of NAACL 2024, pp. 8706-8719. https://aclanthology.org/2024.naacl-long.482/
8. Alzahrani, N., Alyahya, H. A., Alnumay, Y., et al. "When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards." Proceedings of ACL 2024, pp. 13787-13805. https://aclanthology.org/2024.acl-long.744/
9. Wang, Y., Ma, X., Zhang, G., et al. "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark." Advances in Neural Information Processing Systems 37, Datasets and Benchmarks Track, 2024. https://proceedings.neurips.cc/paper_files/paper/2024/hash/ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track.html
10. Zhao, Q., Huang, Y., Lv, T., et al. "MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark." Proceedings of ACL 2025, pp. 13371-13391. https://aclanthology.org/2025.acl-long.656/
11. Singh, S., et al. "Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation." Proceedings of ACL 2025, pp. 18761-18799. https://aclanthology.org/2025.acl-long.919/
12. OpenAI. "Multilingual Massive Multitask Language Understanding (MMMLU)." Hugging Face dataset repository. https://huggingface.co/datasets/openai/MMMLU
13. Li, H., Zhang, Y., Koto, F., et al. "CMMLU: Measuring Massive Multitask Language Understanding in Chinese." Findings of ACL 2024, pp. 11260-11285. https://aclanthology.org/2024.findings-acl.671/
14. Xuan, W., Yang, R., Qi, H., et al. "MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation." Proceedings of EMNLP 2025, pp. 1513-1532. https://aclanthology.org/2025.emnlp-main.79/