Citation and evidence

MMLU-Pro

25 min full readUpdated 8 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

AI BenchmarksLarge Language ModelsMachine Learning

Cite this article

MMLU-Pro
Overview
Full nameMassive Multitask Language Understanding Professional
AbbreviationMMLU-Pro
DescriptionA more robust and challenging multi-task language understanding benchmark with 10-choice questions
Release date2024-06
Latest version1.0
Benchmark updated2024-10
AuthorsYubo Wang, Xueguang Ma, Ge Zhang, Et al.
OrganizationTIGER-AI Lab
Technical Details
TypeKnowledge, Reasoning, Multi-task Understanding
ModalityText
Task formatMultiple choice (up to 10 options)
Number of tasks14 subject categories
Total examples12,032
Evaluation metricAccuracy, Chain-of-Thought performance
Domains14 domains (Biology, Business, Chemistry, Computer Science, Etc.)
LanguagesEnglish
Performance
Human performanceNot reported by the authors
BaselineRandom guess: 10%
SOTA score92.38% (Vals AI)
SOTA modelClaude Fable 5.1
SOTA dateSeptember 2026
SaturatedYes (archived by Vals AI in 2026)
Resources
WebsiteOfficial website
PaperPaper
GitHubRepository
DatasetDownload
LicenseMIT
PredecessorMMLU

MMLU-Pro (Massive Multitask Language Understanding Professional) is an artificial intelligence benchmark of 12,032 ten-choice questions across 14 academic domains, built to measure how well large language models handle hard, reasoning-focused problems. Released in June 2024 by the TIGER-AI Lab, it extends the original MMLU benchmark by adding reasoning-heavy questions, expanding answer choices from four to ten options, and removing trivial or noisy items.[1] The redesign cut model accuracy by 16% to 33% relative to MMLU and reduced score sensitivity to prompt wording from 4-5% down to about 2%, making MMLU-Pro a more discriminative and reproducible test.[1] As of June 2024, the strongest model, GPT-4o, reached 72.6% with Chain-of-Thought prompting, far below its ~88% on MMLU; by September 2026 frontier reasoning models had pushed scores above 92% (Claude Fable 5.1 at 92.38% on Vals AI's tracker), and Vals AI had archived the benchmark as saturated.[5][7] The benchmark was accepted as a Spotlight paper at the NeurIPS 2024 Datasets and Benchmarks Track.[8]

In the words of the authors, MMLU-Pro "not only raises the challenge, causing a significant drop in accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability under varying prompts," making it "a more discriminative benchmark to better track progress in the field."[1]

Overview

MMLU-Pro addresses the performance saturation observed in the original MMLU benchmark, where frontier models had converged to scores between 86-87%, making it difficult to distinguish between model capabilities.[1] By increasing both the difficulty and robustness of questions while reducing prompt sensitivity, MMLU-Pro provides a more challenging and stable evaluation environment for modern language models.[1]

The benchmark was developed by Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen.[1] It was published under the title "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark" and made available on arXiv on June 3, 2024 (arXiv:2406.01574).[1] The dataset is released under the MIT license and hosted on Hugging Face.[4]

What problem does MMLU-Pro solve?

The development of MMLU-Pro was driven by several critical observations about the limitations of the original MMLU benchmark:[1]

  • Performance saturation on original MMLU, with top models clustering at 86-87% accuracy
  • High sensitivity to prompt variations in MMLU (4-5% variance across different prompt styles)
  • Limited discrimination between frontier models, with gaps of only 1-2 percentage points
  • Insufficient emphasis on reasoning versus pure knowledge recall
  • Presence of trivial and noisy questions in the original dataset that inflated scores

The benchmark specifically targets the need for more challenging evaluations that can differentiate between increasingly capable AI systems while providing more stable and reliable measurements. For instance, the gap between GPT-4o and GPT-4-Turbo on MMLU was only about 1%, but MMLU-Pro widened this gap to approximately 9%, providing clearer signal about relative model capabilities.[1]

Technical Specifications

Dataset Composition

MMLU-Pro comprises exactly 12,032 rigorously curated questions spanning 14 diverse domains. Unlike the original MMLU, which drew solely from existing exam questions, MMLU-Pro integrates questions from four distinct sources:[1]

SourceQuestion CountPercentageDescription
Original MMLU6,81056.60%Filtered subset of the original MMLU dataset, with trivial and noisy questions removed
STEM Websites4,08333.93%High-quality STEM problems collected from online educational platforms
TheoremQA5984.97%Human-annotated questions requiring theorem application for resolution
SciBench5414.50%Advanced science questions derived from college-level exams
Total12,032100%

Expanded article table

The inclusion of questions from SciBench and TheoremQA specifically strengthens the STEM coverage of the benchmark. SciBench provides college-level exam questions from physics, chemistry, and mathematics courses, while TheoremQA contributes questions that require applying mathematical and scientific theorems.[1] These additional sources help shift the benchmark's emphasis from factual recall toward multi-step reasoning.[1]

Question Counts by Subject

The 14 subject domains vary in size. The original MMLU's 57 fine-grained subjects were consolidated into 14 broader categories to reduce redundancy and focus on core knowledge areas.[1]

SubjectTotal QuestionsFrom MMLUNewly AddedPrimary Additional Source
Mathematics1,351846505TheoremQA (344), SciBench (161)
Physics1,299411888STEM Websites (617), SciBench (167), TheoremQA (104)
Chemistry1,132178954STEM Websites (741), SciBench (213)
Law1,1011,1010None (entirely from MMLU)
Engineering96967902STEM Websites (902)
Other9249240None (entirely from MMLU)
Economics844444400STEM Websites
Health8188180None (entirely from MMLU)
Psychology798493305STEM Websites
Business789155634STEM Websites (560), TheoremQA (74)
Biology717219498STEM Websites
Philosophy4994990None (entirely from MMLU)
Computer Science410274136TheoremQA (76), STEM Websites (60)
History3813810None (entirely from MMLU)
Total12,0326,8105,222

Expanded article table

Notably, several humanities domains (Law, Philosophy, History, Health) retain 100% of their questions from the original MMLU, while STEM domains received substantial supplementation from external sources. Engineering, for example, draws 93% of its questions from STEM websites rather than from MMLU.[1]

How does MMLU-Pro differ from MMLU?

The following table summarizes the principal design differences between MMLU and MMLU-Pro:

FeatureMMLUMMLU-ProSignificance
Total Questions15,90812,032Quality over quantity
Subject Categories5714Consolidated for clarity
Answer Choices per Question4 (A through D)10 (A through J)Reduces random guessing from 25% to 10%
Random Guess Baseline25%10%60% reduction in guessing advantage
Prompt Sensitivity4-5% variance~2% varianceMore stable and reliable scoring
Question QualityMixed; includes trivial itemsCurated; noise removedStronger signal of true capability
Reasoning RequirementMinimal; mostly knowledge recallSubstantial; multi-step reasoningBetter tests analytical ability
Chain-of-Thought BenefitNegligible or negative+4% to +19% improvementConfirms reasoning-heavy content
Question SourcesExisting exam banks onlyMMLU + SciBench + TheoremQA + STEM sitesBroader, more challenging coverage
PublicationICLR 2021 (Hendrycks et al.)NeurIPS 2024 Datasets Track (Wang et al.)

Expanded article table

Dataset Construction Process

The construction of MMLU-Pro followed a multi-stage pipeline involving automated filtering, GPT-4-based augmentation, and expert human review.[1]

Stage 1: Filtering the Original MMLU

The authors began with the 13,937 questions in the MMLU test set. They evaluated each question against eight language models of varying capability. Questions that were answered correctly by more than four of the eight models were deemed too easy and removed from the dataset. This filtering step eliminated 5,886 questions (approximately 42% of the original set), leaving 8,051 MMLU questions. After additional noise and error removal through expert review, 6,810 MMLU-origin questions were retained for MMLU-Pro.[1]

Stage 2: Integrating STEM Sources

To strengthen coverage in science, technology, engineering, and mathematics, the authors incorporated questions from three additional sources:[1]

  • STEM Websites: 4,083 problems were collected from online educational platforms covering physics, chemistry, engineering, biology, business, economics, computer science, and psychology.
  • TheoremQA: 598 human-annotated questions requiring the application of specific mathematical or scientific theorems.
  • SciBench: 541 college-exam-level questions from physics, chemistry, and mathematics courses.

Questions from the STEM Website (problem statements with solutions) and TheoremQA (questions with brief answers) were not in multiple-choice format. GPT-4-Turbo was used to extract short correct answers and to generate three distractors for each, and the extracted answers were then checked manually against the solutions.[1]

Stage 3: Expanding to Ten Options

One of the most distinctive features of MMLU-Pro is its expansion from four answer choices to ten. For questions that originally had only four options, GPT-4-Turbo was employed to generate six additional plausible distractors for each question. These distractors were not random; they were designed to be plausible incorrect answers that require discriminative reasoning to eliminate. This expansion reduces the probability of guessing correctly from 25% (with four choices) to 10% (with ten choices), placing greater demand on genuine understanding.[1]

Stage 4: Expert Review

The dataset underwent two phases of expert review:[1]

  1. Phase 1 (Correctness Verification): Experts verified each answer, removed questions unsuitable for a multiple-choice format, and discarded questions that lacked necessary information or required images or tables.
  2. Phase 2 (Distractor Validation): Gemini 1.5 Pro was used to re-evaluate all answer options and flag potential "false negatives" (distractors that might actually be correct). Human experts then reviewed each flagged case to confirm that distractors were genuinely incorrect and sufficiently distinct from the correct answer.

Evaluation Methodology

Answer Format

Each question in MMLU-Pro follows a standardized format:

  • Question Stem: The main question or problem statement
  • Up to 10 Answer Options: Labeled A through J (83% of questions have all ten; the average is 9.47)
  • Single Correct Answer: Exactly one correct option per question
  • Distractors: Up to nine plausible but incorrect options

Scoring System

MetricDescriptionCalculation
Overall AccuracyPercentage of correct answers across all questions(Correct / Total) x 100%
Domain AccuracyPerformance within each of the 14 subject areas(Correct in domain / Total in domain) x 100%
CoT AccuracyAccuracy when the model uses Chain-of-Thought promptingCoT correct / Total x 100%
Direct Answer AccuracyAccuracy with direct answer extraction (no reasoning)Direct correct / Total x 100%
CoT GainImprovement attributable to Chain-of-Thought reasoningCoT accuracy minus Direct accuracy

Expanded article table

The standard evaluation protocol uses 5-shot Chain-of-Thought prompting, where the model is given five example questions with worked-out reasoning before being asked to solve test questions.[1]

Prompt Robustness

A key advantage of MMLU-Pro over its predecessor is reduced sensitivity to prompt wording. The authors evaluated models under 24 different but reasonable prompts. On MMLU the prompt-induced score range was generally 4-5%, with peaks up to 10.98%; on MMLU-Pro it was generally around 2%, with a maximum of 3.74%.[1] The 24 prompt styles were added to the GitHub repository in October 2024.[3]

On the original MMLU, the same prompt variations produced 4-5% swings in model scores. MMLU-Pro's reduced sensitivity (approximately 2% total variance) means that benchmark results are more reproducible and less dependent on prompt engineering choices.[1]

Chain-of-Thought vs. Direct Answering

One of the most important findings from the MMLU-Pro paper is the stark difference in how Chain-of-Thought (CoT) reasoning affects performance compared to the original MMLU.[1]

On the original MMLU, CoT prompting provided minimal or even slightly negative benefit. This was because most MMLU questions tested factual recall rather than multi-step reasoning, and the additional reasoning steps sometimes introduced errors without improving accuracy. On MMLU-Pro, however, CoT prompting improves every model in the paper's comparison, by 3.9 points (Llama-3-8B) up to 19.1 points (GPT-4o), confirming that the benchmark genuinely requires reasoning rather than simple knowledge lookup.[1]

CoT vs. Direct Answer Comparison

ModelMMLU (CoT)MMLU (Direct)MMLU CoT GainMMLU-Pro (CoT)MMLU-Pro (Direct)MMLU-Pro CoT Gain
GPT-4o88.7%87.2%+1.5%72.6%53.5%+19.1%
GPT-4-Turbo86.5%86.7%-0.2%63.7%48.4%+15.3%
Phi-3-medium79.4%78.0%+1.4%55.7%47.5%+8.2%

Expanded article table

The CoT gain on MMLU-Pro is dramatically larger than on MMLU. GPT-4o, for example, improves by 19.1 percentage points when using Chain-of-Thought on MMLU-Pro, compared to only 1.5 points on MMLU. GPT-4-Turbo actually performs slightly worse with CoT on the original MMLU (-0.2%), but gains 15.3 points on MMLU-Pro. These results demonstrate that MMLU-Pro questions genuinely benefit from step-by-step reasoning and cannot be answered through pattern matching alone.[1]

Performance Analysis

Original Paper Results (June 2024)

The following results were reported in the original MMLU-Pro paper using CoT evaluation (5-shot, except 0-shot for the two Gemini 1.5 models). These scores represent model capabilities at the time of the benchmark's release.[1]

RankModelTypeMMLU-Pro Score (CoT)Organization
1GPT-4oClosed-source72.6%OpenAI
2Gemini 1.5 ProClosed-source69.0%Google
3Claude 3 OpusClosed-source68.5%Anthropic
4GPT-4-TurboClosed-source63.7%OpenAI
5Gemini 1.5 FlashClosed-source59.1%Google
6Yi-largeClosed-source58.1%01.AI
7Claude 3 SonnetClosed-source56.8%Anthropic
8LLaMA 3 70B InstructOpen-source56.2%Meta
9Phi-3-medium-4kOpen-source55.7%Microsoft
10DeepSeek V2 ChatOpen-source54.8%DeepSeek
11LLaMA 3 70BOpen-source52.8%Meta
12Qwen 1.5 72B ChatOpen-source52.6%Alibaba
13Yi-1.5-34B-ChatOpen-source52.3%01.AI
14Phi-3-medium-128k (repository leaderboard, not in the paper)Open-source51.9%Microsoft
15MAmmoTH2-8x7B-PlusOpen-source50.4%TIGER-AI Lab
16Mixtral 8x7B InstructOpen-source43.3%Mistral AI

Expanded article table

At the time of publication, GPT-4o was the clear leader at 72.6%. Open-source models lagged behind closed-source models by a significant margin, with the best open-source model (LLaMA 3 70B Instruct) scoring 16.4 percentage points below GPT-4o.[1]

Later results (as of September 2026)

Since the benchmark's release, newer models have achieved substantially higher scores. The following table shows the top of Vals AI's MMLU-Pro leaderboard as last updated on September 1, 2026, plus selected older models for context. Scores from different trackers are not directly comparable because prompts and settings differ.[7]

Vals AI rankModelMMLU-Pro ScoreOrganization
1Claude Fable 5.192.38%Anthropic
2Claude Opus 591.59%Anthropic
3Claude Fable 591.50%Anthropic
4Gemini 3.1 Pro Preview90.99%Google
5Gemini 3.8 Flash90.22%Google
6Gemini 3.7 Flash90.12%Google
7Gemini 3 Pro90.10%Google
8Claude Opus 4.789.87%Anthropic
9Claude Opus 4.889.58%Anthropic
10Gemini 3.5 Flash89.52%Google
21GPT-5.588.14%OpenAI
31Claude Opus 4.5 (Thinking)87.26%Anthropic
56o385.59%OpenAI
75DeepSeek R183.18%DeepSeek

Expanded article table

By March 2026 the official TIGER-Lab leaderboard listed about 260 entries, led by Gemini 3.1 Pro at 91.16% in TIGER-Lab's own evaluation, and Artificial Analysis had moved MMLU-Pro to its legacy evaluations, where Gemini 3 Pro Preview (High) led its runs at 89.8%.[5][6] Frontier models such as Gemini 3 Pro and Claude Opus 4.5 had already passed 87% by late 2025. On Vals AI's tracker the top three models in September 2026 sat within 0.9 points of each other, and Vals AI archived the benchmark, saying that performance had saturated.[7]

Accuracy Drop from MMLU to MMLU-Pro

Models experience significant accuracy reductions when evaluated on MMLU-Pro compared to their MMLU performance. The paper reports an overall accuracy drop of 16% to 33% relative to MMLU, and the magnitude of the drop correlates with model capability: stronger models tend to lose fewer percentage points.[1]

ModelMMLU (CoT)MMLU-Pro (CoT)Drop (points)
GPT-4o88.7%72.6%16.1
GPT-4-Turbo86.5%63.7%22.8
Phi-3-medium-4k-instruct79.4%55.7%23.7
Llama-3-8B62.7%35.4%27.3
Gemma-7B62.4%33.7%28.7

Expanded article table

The paired scores above are from Table 3 of the paper.[1]

The consistent pattern of larger drops for weaker models confirms that MMLU-Pro is more discriminative. On MMLU, a mid-range model might appear to be within striking distance of a frontier model, but MMLU-Pro reveals a much wider gap in actual capability.[1]

Error Analysis

The MMLU-Pro paper includes a detailed error analysis of 120 incorrect responses from GPT-4o, which was the top-performing model at the time of publication. The analysis categorizes errors into four types:[1]

Error TypePercentageDescription
Reasoning Errors39%Logical inconsistencies or flawed inference chains, even when the model recalled relevant knowledge correctly
Knowledge Gaps35%Missing domain-specific expertise needed to arrive at the correct answer
Calculation Mistakes12%Arithmetic or computational errors in problems requiring numerical work
Other Errors14%No selection made (5%), question understanding errors (4%), generation issues (2%), annotation errors (2%), and answer extraction errors (1%)

Expanded article table

The dominance of reasoning errors (39%) over knowledge gaps (35%) is noteworthy. It indicates that MMLU-Pro successfully tests reasoning ability rather than merely factual knowledge. Models frequently recalled the correct principles and formulas but failed to apply them through a valid chain of logical steps. The paper files incorrect financial calculations and misapplied optical principles under knowledge gaps rather than reasoning errors, and separately notes that in calculation errors the model often had the correct formula but computed the values wrongly.[1]

Domain-Specific Performance

Subject Area Analysis

Table 2 of the paper reports per-subject CoT accuracy for six representative subjects. GPT-4o, the top model, is shown with the best open model of the time, Llama-3-70B-Instruct:[1]

DomainGPT-4oLlama-3-70B-Instruct
Mathematics76.1%54.0%
Physics74.7%49.6%
Engineering55.0%43.6%
History70.1%56.9%
Law51.0%39.9%
Psychology79.2%70.2%

Expanded article table

The paper singles out Engineering and Law as the subjects that "consistently scored lower." It attributes the Engineering results to new STEM Website questions "which require complex formula derivations and multi-step calculations," and says "Law scores suffer as questions become more intricate and detailed with additional options." Math and Physics show the widest spread between models, from over 70% for GPT-4o to just over 20% for Mistral-7B-v0.1.[1]

Why is MMLU-Pro harder than MMLU?

Several design choices combine to make MMLU-Pro substantially more difficult than the original MMLU:

More Distractors

With ten answer options instead of four, models face three times as many distractors (nine instead of three). Each additional distractor is not random filler but a carefully constructed incorrect answer that a model with partial understanding might select. The probability of guessing correctly drops from 25% to 10%, which means that a model's score above the 10% baseline more directly reflects genuine comprehension.[1]

Reasoning-Intensive Questions

The filtering process removed questions that most models could answer through simple recall. The remaining questions, combined with new additions from SciBench and TheoremQA, demand multi-step reasoning. A typical MMLU-Pro question in physics or mathematics may require setting up equations, performing algebraic manipulation, applying domain-specific theorems, and then selecting from ten closely spaced numerical or conceptual answers.[1]

Higher-Quality Questions

By removing trivial and noisy items from the original MMLU and adding questions from vetted academic sources, MMLU-Pro achieves a higher floor of question quality. Each question meaningfully tests competence rather than rewarding surface-level pattern recognition.[1]

Reduced Exploitability

The combination of ten options and reduced prompt sensitivity makes MMLU-Pro harder to game. On the original MMLU, certain prompt formulations could swing scores by 4-5 percentage points, creating an incentive to optimize prompts rather than improve model capability. MMLU-Pro's ~2% prompt sensitivity largely eliminates this source of variance.[1]

Implementation

Installation and Setup

# Clone the repository
git clone https://github.com/TIGER-AI-Lab/MMLU-Pro
cd MMLU-Pro

# Install dependencies
pip install -r requirements.txt

Running Evaluations

The repository README provides example shell scripts for local and API inference; the dataset itself is pulled from Hugging Face.[3]

# Local inference (edit the model name in the script first)
cd scripts/examples/
sh eval_llama_2_7b.sh

# API inference (set the API key in evaluate_from_api.py first)
cd scripts/examples/
sh eval_gpt_4.sh

Dataset Access

The dataset is available on Hugging Face and can be loaded directly using the datasets library:[4]

from datasets import load_dataset

# Load MMLU-Pro dataset
dataset = load_dataset("TIGER-Lab/MMLU-Pro")

# Access specific split
test_set = dataset['test']       # 12,032 rows
validation_set = dataset['validation']  # 70 rows

# Filter by domain
math_questions = dataset['test'].filter(lambda x: x['category'] == 'math')

Each record in the dataset includes the following fields:[4]

FieldTypeDescription
question_idint64Unique identifier for the question
questionstringThe question text
optionslist of stringsTen multiple-choice options
answerstringCorrect answer letter (A through J)
answer_indexint64Index of correct answer (0 through 9)
cot_contentstringReference Chain-of-Thought reasoning
categorystringSubject category (one of 14 domains)
srcstringSource of the question

Expanded article table

Adoption and Impact

Since its release in June 2024, MMLU-Pro has rapidly become a standard evaluation benchmark in the large language model community.

Who uses MMLU-Pro?

Major AI labs now routinely report MMLU-Pro scores alongside other benchmarks when releasing new models. OpenAI, Anthropic, Google DeepMind, Meta, Mistral AI, and DeepSeek have all published MMLU-Pro results for their flagship models. Independent benchmark platforms including Artificial Analysis and Vals AI tracked MMLU-Pro scores, though by 2026 Artificial Analysis listed it among its legacy evaluations and Vals AI had archived it as saturated.[6][7]

Research Influence

The MMLU-Pro paper has been widely cited in subsequent research on LLM evaluation methodology. It has also inspired derivative benchmarks:

  • MMLU-ProX (published at EMNLP 2025): Translates MMLU-Pro into 29 typologically diverse languages, enabling cross-lingual evaluation of multilingual models.
  • Mobile-MMLU: Adapts the multitask evaluation format to mobile and edge computing scenarios, expanding topical coverage to 80 mobile-relevant domains.

Approaching Saturation

With frontier reasoning models scoring above 92% on MMLU-Pro by September 2026, the benchmark faces the same saturation problem that originally motivated its creation; Vals AI stopped running it on new models for that reason.[7] The gap between top models has narrowed, and further improvements may be difficult to measure reliably. This trend suggests that even more challenging benchmarks may be needed in the future to differentiate the next generation of models.

Challenges and Insights

Key Challenges for Models

ChallengeDescriptionImpact
Distractor QualityHigh-quality incorrect options generated by GPT-4-TurboReduces guessing success and requires precise discrimination
Reasoning DepthMulti-step problem solving required across STEM and non-STEM domainsChallenges surface-level knowledge
Domain ExpertiseSpecialized knowledge needed in 14 distinct domainsTests both breadth and depth of training data
Option DiscriminationSubtle differences between closely related choicesRequires precise understanding rather than approximate matching
Cross-Domain IntegrationSome questions require combining knowledge from multiple fieldsTests holistic understanding

Expanded article table

Common Failure Modes

  1. Surface Pattern Matching: Selecting answers based on superficial similarities rather than genuine understanding
  2. Incomplete Reasoning: Beginning a valid reasoning chain but stopping before reaching the correct conclusion
  3. Domain Confusion: Applying knowledge or methods from the wrong field to a given problem
  4. Distractor Attraction: Being misled by plausible but incorrect options, especially those that represent common misconceptions
  5. Calculation Errors: Performing arithmetic or algebraic steps incorrectly even when the overall approach is sound

Limitations and Considerations

Current Limitations

LimitationDescriptionImpact
English OnlyAll questions are in EnglishLimits applicability for non-English language models (partially addressed by MMLU-ProX)
Multiple Choice FormatAll questions use multiple choice with up to 10 optionsMay not capture free-form reasoning or generation capabilities
Static DatasetFixed set of 12,032 questionsCreates risk of contamination if questions appear in training data
Academic FocusQuestions are drawn from academic exams and textbooksMay not reflect practical, real-world problem-solving ability
SaturationFrontier models exceed 92% (Vals AI, September 2026)Diminishing ability to differentiate top-tier models
GPT-4 Bias in DistractorsAdditional options were generated by GPT-4-TurboModels from the same family could be affected, although the authors report that GPT-4-Turbo gained no advantage from the augmentation

Expanded article table

Future Directions

  1. Multilingual Extension: MMLU-ProX covers 29 languages; further expansion is ongoing
  2. Dynamic Generation: Procedurally generated questions to prevent training data contamination
  3. Free-Form Responses: Open-ended answer formats to test generation rather than selection
  4. Multimodal Integration: Adding visual and audio components to test cross-modal reasoning
  5. Adaptive Testing: Adjusting question difficulty based on model performance during evaluation
  6. Harder Benchmarks: Projects like GPQA and HLE (Humanity's Last Exam) target even higher difficulty ceilings
  • MMLU: Original Massive Multitask Language Understanding (Hendrycks et al., 2021)
  • GPQA: Graduate-level science questions requiring PhD-level expertise
  • ARC: AI2 Reasoning Challenge for grade-school science questions
  • HellaSwag: Commonsense reasoning about everyday situations
  • BigBench: Diverse capability evaluation across 200+ tasks
  • AGIEval: Evaluation using human-level standardized exams
  • MATH: Competition-level mathematics problem solving
  • TheoremQA: Theorem-application questions (also a source for MMLU-Pro)
  • GSM8K: Grade-school math word problems

Significance

MMLU-Pro represents a crucial evolution in language model evaluation, addressing the saturation problem of its predecessor while providing more robust and discriminative assessment. Its reduced prompt sensitivity and enhanced focus on reasoning make it particularly valuable for:

  • Distinguishing between frontier model capabilities at a finer granularity than MMLU allows
  • Tracking genuine progress in AI reasoning development over time
  • Identifying whether models rely on reasoning versus memorization
  • Providing stable, reproducible performance measurements across different evaluation setups
  • Guiding model development priorities by highlighting specific domains and error types

The benchmark's success in revealing performance differences previously hidden by MMLU's ceiling effect has made it one of the most widely used LLM evaluation tools since its introduction. Its acceptance as a Spotlight paper at NeurIPS 2024 further cemented its standing in the research community.[8]

Continue reading

More from See also

More to read

Topics:AI BenchmarksLarge Language ModelsMachine Learning

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., & Chen, W. (2024). "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark." *Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), Datasets and Benchmarks Track*. arXiv:2406.01574
  2. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). "Measuring Massive Multitask Language Understanding." *Proceedings of the International Conference on Learning Representations (ICLR 2021)*. arXiv:2009.03300
  3. ^1 ^2TIGER-AI Lab. "MMLU-Pro GitHub Repository." github.com/...MMLU-Pro
  4. ^1 ^2 ^3TIGER-AI Lab. "MMLU-Pro Dataset on Hugging Face." huggingface.co/...MMLU-Pro
  5. ^1 ^2TIGER-AI Lab. "MMLU-Pro Leaderboard." huggingface.co/...MMLU-Pro
  6. ^1 ^2Artificial Analysis. "MMLU-Pro Benchmark Leaderboard." artificialanalysis.ai/...mmlu-pro
  7. ^1 ^2 ^3 ^4 ^5Vals AI. "MMLU Pro Benchmark." vals.ai/...mmlu_pro
  8. ^1 ^2NeurIPS 2024. "Poster: MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark." neurips.cc/...97435

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

10 revisions · v10 · 5,067 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check of the full article against its cited sources (150 claims, 14 sources fetched); 13 material + 21 minor fixed

Cite this page: AI Wiki. "MMLU-Pro." aiwiki.ai, updated 21 Jun 2026, fact-checked 2 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/mmlu-pro

Edit article