MMLU-Pro

MMLU-Pro
Overview
Full name	Massive Multitask Language Understanding Professional
Abbreviation	MMLU-Pro
Description	A more robust and challenging multi-task language understanding benchmark with 10-choice questions
Release date	2024-06
Latest version	1.0
Benchmark updated	2024-10
Authors	Yubo Wang, Xueguang Ma, Ge Zhang, Et al.
Organization	TIGER-AI Lab
Technical Details
Type	Knowledge, Reasoning, Multi-task Understanding
Modality	Text
Task format	Multiple choice (10 options)
Number of tasks	12,032
Total examples	12,032
Evaluation metric	Accuracy, Chain-of-Thought performance
Domains	14 domains (Biology, Business, Chemistry, Computer Science, Etc.)
Languages	English
Performance
Human performance	~90% (estimated)
Baseline	Random guess: 10%
SOTA score	~90.1%
SOTA model	Gemini 3 Pro
SOTA date	2025
Saturated	Approaching
Resources
Website	Official website
Paper	Paper
GitHub	Repository
Dataset	Download
License	MIT
Predecessor	MMLU

MMLU-Pro (Massive Multitask Language Understanding Professional) is an artificial intelligence benchmark of 12,032 ten-choice questions across 14 academic domains, built to measure how well large language models handle hard, reasoning-focused problems. Released in June 2024 by the TIGER-AI Lab, it extends the original MMLU benchmark by adding reasoning-heavy questions, expanding answer choices from four to ten options, and removing trivial or noisy items.^[1] The redesign cut model accuracy by 16% to 33% relative to MMLU and reduced score sensitivity to prompt wording from 4-5% down to about 2%, making MMLU-Pro a more discriminative and reproducible test.^[1] As of June 2024, the strongest model, GPT-4o, reached 72.6% with Chain-of-Thought prompting, far below its ~88% on MMLU; by 2026 frontier reasoning models such as Gemini 3 Pro had pushed scores to roughly 90.1%, approaching the estimated human ceiling.^[1][5][6] The benchmark was accepted as a Spotlight paper at the NeurIPS 2024 Datasets and Benchmarks Track.^[8]

In the words of the authors, MMLU-Pro "not only raises the challenge, causing a significant drop in accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability under varying prompts," making it "a more discriminative benchmark to better track progress in the field."^[1]

Overview

MMLU-Pro addresses the performance saturation observed in the original MMLU benchmark, where frontier models had converged to scores between 86-87%, making it difficult to distinguish between model capabilities.^[1] By increasing both the difficulty and robustness of questions while reducing prompt sensitivity, MMLU-Pro provides a more challenging and stable evaluation environment for modern language models.^[1]

The benchmark was developed by Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen.^[1] It was published under the title "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark" and made available on arXiv on June 3, 2024 (arXiv:2406.01574).^[1] The dataset is released under the MIT license and hosted on Hugging Face.^[4]

What problem does MMLU-Pro solve?

The development of MMLU-Pro was driven by several critical observations about the limitations of the original MMLU benchmark:^[1]

• Performance saturation on original MMLU, with top models clustering at 86-87% accuracy
• High sensitivity to prompt variations in MMLU (4-5% variance across different prompt styles)
• Limited discrimination between frontier models, with gaps of only 1-2 percentage points
• Insufficient emphasis on reasoning versus pure knowledge recall
• Presence of trivial and noisy questions in the original dataset that inflated scores

The benchmark specifically targets the need for more challenging evaluations that can differentiate between increasingly capable AI systems while providing more stable and reliable measurements. For instance, the gap between GPT-4o and GPT-4-Turbo on MMLU was only about 1%, but MMLU-Pro widened this gap to approximately 9%, providing clearer signal about relative model capabilities.^[1]

Technical Specifications

Dataset Composition

MMLU-Pro comprises exactly 12,032 rigorously curated questions spanning 14 diverse domains. Unlike the original MMLU, which drew solely from existing exam questions, MMLU-Pro integrates questions from four distinct sources:^[1]

Scroll sideways for more →

The inclusion of questions from SciBench and TheoremQA specifically strengthens the STEM coverage of the benchmark. SciBench provides college-level exam questions from physics, chemistry, and mathematics courses, while TheoremQA contributes questions that require applying mathematical and scientific theorems.^[1] These additional sources help shift the benchmark's emphasis from factual recall toward multi-step reasoning.^[1]

Question Counts by Subject

The 14 subject domains vary in size. The original MMLU's 57 fine-grained subjects were consolidated into 14 broader categories to reduce redundancy and focus on core knowledge areas.^[1]

Scroll sideways for more →

Notably, several humanities domains (Law, Philosophy, History, Health) retain 100% of their questions from the original MMLU, while STEM domains received substantial supplementation from external sources. Engineering, for example, draws 93% of its questions from STEM websites rather than from MMLU.^[1]

How does MMLU-Pro differ from MMLU?

The following table summarizes the principal design differences between MMLU and MMLU-Pro:

Scroll sideways for more →

Dataset Construction Process

The construction of MMLU-Pro followed a multi-stage pipeline involving automated filtering, GPT-4-based augmentation, and expert human review.^[1]

Stage 1: Filtering the Original MMLU

The authors began with the 13,937 questions in the MMLU test set. They evaluated each question against eight language models of varying capability. Questions that were answered correctly by four or more of the eight models were deemed too easy and removed from the dataset. This filtering step eliminated 5,886 questions (approximately 42% of the original set), leaving 8,051 MMLU questions. After additional noise and error removal through expert review, 6,810 MMLU-origin questions were retained for MMLU-Pro.^[1]

Stage 2: Integrating STEM Sources

To strengthen coverage in science, technology, engineering, and mathematics, the authors incorporated questions from three additional sources:^[1]

• STEM Websites: 4,083 problems were collected from online educational platforms covering physics, chemistry, engineering, biology, business, economics, computer science, and psychology.
• TheoremQA: 598 human-annotated questions requiring the application of specific mathematical or scientific theorems.
• SciBench: 541 college-exam-level questions from physics, chemistry, and mathematics courses.

Questions from TheoremQA and SciBench were originally in free-response format. GPT-4-Turbo was used to extract short correct answers from the solutions and convert them into multiple-choice format.^[1]

Stage 3: Expanding to Ten Options

One of the most distinctive features of MMLU-Pro is its expansion from four answer choices to ten. For questions that originally had only four options, GPT-4-Turbo was employed to generate six additional plausible distractors for each question. These distractors were not random; they were designed to be plausible incorrect answers that require discriminative reasoning to eliminate. This expansion reduces the probability of guessing correctly from 25% (with four choices) to 10% (with ten choices), placing greater demand on genuine understanding.^[1]

Stage 4: Expert Review

The dataset underwent two phases of expert review:^[1]

1. Phase 1 (Correctness Verification): A panel of more than ten domain experts reviewed questions and their correct answers for factual accuracy.
2. Phase 2 (Distractor Validation): Gemini 1.5 Pro was used to re-evaluate all answer options and flag potential "false negatives" (distractors that might actually be correct). Human experts then reviewed each flagged case to confirm that distractors were genuinely incorrect and sufficiently distinct from the correct answer.

Evaluation Methodology

Answer Format

Each question in MMLU-Pro follows a standardized format:

• Question Stem: The main question or problem statement
• 10 Answer Options: Labeled A through J
• Single Correct Answer: Exactly one correct option per question
• Distractors: Nine plausible but incorrect options

Scoring System

Scroll sideways for more →

The standard evaluation protocol uses 5-shot Chain-of-Thought prompting, where the model is given five example questions with worked-out reasoning before being asked to solve test questions.^[1]

Prompt Robustness

A key advantage of MMLU-Pro over its predecessor is reduced sensitivity to prompt wording. The authors tested 24 different prompt styles across four categories:^[1]

Scroll sideways for more →

On the original MMLU, the same prompt variations produced 4-5% swings in model scores. MMLU-Pro's reduced sensitivity (approximately 2% total variance) means that benchmark results are more reproducible and less dependent on prompt engineering choices.^[1]

Chain-of-Thought vs. Direct Answering

One of the most important findings from the MMLU-Pro paper is the stark difference in how Chain-of-Thought (CoT) reasoning affects performance compared to the original MMLU.^[1]

On the original MMLU, CoT prompting provided minimal or even slightly negative benefit. This was because most MMLU questions tested factual recall rather than multi-step reasoning, and the additional reasoning steps sometimes introduced errors without improving accuracy. On MMLU-Pro, however, CoT prompting yields large and consistent improvements across all models tested, confirming that the benchmark genuinely requires reasoning rather than simple knowledge lookup.^[1]

CoT vs. Direct Answer Comparison

Scroll sideways for more →

The CoT gain on MMLU-Pro is dramatically larger than on MMLU. GPT-4o, for example, improves by 19.1 percentage points when using Chain-of-Thought on MMLU-Pro, compared to only 1.5 points on MMLU. GPT-4-Turbo actually performs slightly worse with CoT on the original MMLU (-0.2%), but gains 15.3 points on MMLU-Pro. These results demonstrate that MMLU-Pro questions genuinely benefit from step-by-step reasoning and cannot be answered through pattern matching alone.^[1]

The subjects showing the largest CoT improvements include Mathematics (+41.8 percentage points for some models), Chemistry (+39.5 points), and Business (+39.4 points), indicating that these domains contain the most reasoning-intensive questions.^[1]

Performance Analysis

Original Paper Results (June 2024)

The following results were reported in the original MMLU-Pro paper using 5-shot CoT evaluation. These scores represent model capabilities at the time of the benchmark's release.^[1]

Scroll sideways for more →

At the time of publication, GPT-4o was the clear leader at 72.6%. Open-source models lagged behind closed-source models by a significant margin, with the best open-source model (LLaMA 3 70B Instruct) scoring 16.4 percentage points below GPT-4o.^[1]

Updated Leaderboard (as of early 2026)

Since the benchmark's release, newer models have achieved substantially higher scores. The following table reflects reported MMLU-Pro scores from the official leaderboard and third-party evaluation platforms.^[5][6]

Scroll sideways for more →

As of early 2026, 199 models have been evaluated on MMLU-Pro, with an average score of 74.4% across all submissions.^[6] Frontier models with extended reasoning capabilities (such as Claude Opus 4.5 Thinking and Gemini 3 Pro) have pushed scores above 89%, approaching the estimated human performance ceiling of approximately 90%. Independent trackers in 2026 reported the top models clustered within roughly one percentage point of each other, a clear sign that the benchmark is nearing saturation for the strongest systems.^[5][6]

Accuracy Drop from MMLU to MMLU-Pro

Models experience significant accuracy reductions when evaluated on MMLU-Pro compared to their MMLU performance. The paper reports an overall accuracy drop of 16% to 33% relative to MMLU, and the magnitude of the drop correlates with model capability: stronger models tend to lose fewer percentage points.^[1]

Scroll sideways for more →

The consistent pattern of larger drops for weaker models confirms that MMLU-Pro is more discriminative. On MMLU, a mid-range model might appear to be within striking distance of a frontier model, but MMLU-Pro reveals a much wider gap in actual capability.^[1]

Error Analysis

The MMLU-Pro paper includes a detailed error analysis of 120 incorrect responses from GPT-4o, which was the top-performing model at the time of publication. The analysis categorizes errors into four types:^[1]

Scroll sideways for more →

The dominance of reasoning errors (39%) over knowledge gaps (35%) is noteworthy. It indicates that MMLU-Pro successfully tests reasoning ability rather than merely factual knowledge. Models frequently recalled the correct principles and formulas but failed to apply them through a valid chain of logical steps. For example, GPT-4o sometimes used incorrect values in financial calculations or misapplied refractive index ratios in optics problems, despite demonstrating awareness of the relevant concepts.^[1]

Domain-Specific Performance

Subject Area Analysis

Scroll sideways for more →

STEM subjects (Mathematics, Physics, Chemistry, Engineering) consistently rank as the most difficult, which aligns with the heavy supplementation of these domains with questions from SciBench, TheoremQA, and STEM websites. Law also ranks as particularly challenging despite containing only questions from the original MMLU, likely because legal reasoning requires nuanced interpretation of multiple overlapping rules.^[1]

Why is MMLU-Pro harder than MMLU?

Several design choices combine to make MMLU-Pro substantially more difficult than the original MMLU:

More Distractors

With ten answer options instead of four, models face 2.5 times as many plausible alternatives. Each additional distractor is not random filler but a carefully constructed incorrect answer that a model with partial understanding might select. The probability of guessing correctly drops from 25% to 10%, which means that a model's score above the 10% baseline more directly reflects genuine comprehension.^[1]

Reasoning-Intensive Questions

The filtering process removed questions that most models could answer through simple recall. The remaining questions, combined with new additions from SciBench and TheoremQA, demand multi-step reasoning. A typical MMLU-Pro question in physics or mathematics may require setting up equations, performing algebraic manipulation, applying domain-specific theorems, and then selecting from ten closely spaced numerical or conceptual answers.^[1]

Higher-Quality Questions

By removing trivial and noisy items from the original MMLU and adding questions from vetted academic sources, MMLU-Pro achieves a higher floor of question quality. Each question meaningfully tests competence rather than rewarding surface-level pattern recognition.^[1]

Reduced Exploitability

The combination of ten options and reduced prompt sensitivity makes MMLU-Pro harder to game. On the original MMLU, certain prompt formulations could swing scores by 4-5 percentage points, creating an incentive to optimize prompts rather than improve model capability. MMLU-Pro's ~2% prompt sensitivity largely eliminates this source of variance.^[1]

Implementation

Installation and Setup

# Clone the repository
git clone https://github.com/TIGER-AI-Lab/MMLU-Pro
cd MMLU-Pro

# Install dependencies
pip install -r requirements.txt

# Download dataset
python download_data.py

Running Evaluations

# Basic evaluation
python evaluate.py --model "gpt-4" --method "direct"

# With Chain-of-Thought
python evaluate.py --model "gpt-4" --method "cot"

# Specific domains
python evaluate.py --model "gpt-4" --domains "math,physics,cs"

# All 24 prompt styles
python evaluate.py --model "gpt-4" --test-prompts

Dataset Access

The dataset is available on Hugging Face and can be loaded directly using the datasets library:^[4]

from datasets import load_dataset

# Load MMLU-Pro dataset
dataset = load_dataset("TIGER-Lab/MMLU-Pro")

# Access specific split
test_set = dataset['test']       # 12,000 rows
validation_set = dataset['validation']  # 70 rows

# Filter by domain
math_questions = dataset['test'].filter(lambda x: x['category'] == 'math')

Each record in the dataset includes the following fields:^[4]

Scroll sideways for more →

Adoption and Impact

Since its release in June 2024, MMLU-Pro has rapidly become a standard evaluation benchmark in the large language model community.

Who uses MMLU-Pro?

Major AI labs now routinely report MMLU-Pro scores alongside other benchmarks when releasing new models. OpenAI, Anthropic, Google DeepMind, Meta, Mistral AI, and DeepSeek have all published MMLU-Pro results for their flagship models. Multiple independent benchmark aggregation platforms, including Artificial Analysis, Vals AI, and Kaggle Benchmarks, track MMLU-Pro scores, and as of early 2026, 199 models have been evaluated on the benchmark.^[6][7]

Research Influence

The MMLU-Pro paper has been widely cited in subsequent research on LLM evaluation methodology. It has also inspired derivative benchmarks:

• MMLU-ProX (published at EMNLP 2025): Translates MMLU-Pro into 29 typologically diverse languages, enabling cross-lingual evaluation of multilingual models.
• Mobile-MMLU: Adapts the multitask evaluation format to mobile and edge computing scenarios, expanding topical coverage to 80 mobile-relevant domains.

Approaching Saturation

With frontier reasoning models now scoring above 89% on MMLU-Pro, the benchmark is beginning to face the same saturation challenge that originally motivated its creation.^[6] The gap between top models has narrowed, and further improvements may be difficult to measure reliably. This trend suggests that even more challenging benchmarks may be needed in the future to differentiate the next generation of models.

Challenges and Insights

Key Challenges for Models

Scroll sideways for more →

Common Failure Modes

1. Surface Pattern Matching: Selecting answers based on superficial similarities rather than genuine understanding
2. Incomplete Reasoning: Beginning a valid reasoning chain but stopping before reaching the correct conclusion
3. Domain Confusion: Applying knowledge or methods from the wrong field to a given problem
4. Distractor Attraction: Being misled by plausible but incorrect options, especially those that represent common misconceptions
5. Calculation Errors: Performing arithmetic or algebraic steps incorrectly even when the overall approach is sound

Limitations and Considerations

Current Limitations

Scroll sideways for more →

Future Directions

1. Multilingual Extension: MMLU-ProX covers 29 languages; further expansion is ongoing
2. Dynamic Generation: Procedurally generated questions to prevent training data contamination
3. Free-Form Responses: Open-ended answer formats to test generation rather than selection
4. Multimodal Integration: Adding visual and audio components to test cross-modal reasoning
5. Adaptive Testing: Adjusting question difficulty based on model performance during evaluation
6. Harder Benchmarks: Projects like GPQA and HLE (Humanity's Last Exam) target even higher difficulty ceilings

Related Benchmarks

• MMLU: Original Massive Multitask Language Understanding (Hendrycks et al., 2021)
• GPQA: Graduate-level science questions requiring PhD-level expertise
• ARC: AI2 Reasoning Challenge for grade-school science questions
• HellaSwag: Commonsense reasoning about everyday situations
• BigBench: Diverse capability evaluation across 200+ tasks
• AGIEval: Evaluation using human-level standardized exams
• MATH: Competition-level mathematics problem solving
• TheoremQA: Theorem-application questions (also a source for MMLU-Pro)
• GSM8K: Grade-school math word problems

Significance

MMLU-Pro represents a crucial evolution in language model evaluation, addressing the saturation problem of its predecessor while providing more robust and discriminative assessment. Its reduced prompt sensitivity and enhanced focus on reasoning make it particularly valuable for:

• Distinguishing between frontier model capabilities at a finer granularity than MMLU allows
• Tracking genuine progress in AI reasoning development over time
• Identifying whether models rely on reasoning versus memorization
• Providing stable, reproducible performance measurements across different evaluation setups
• Guiding model development priorities by highlighting specific domains and error types

The benchmark's success in revealing performance differences previously hidden by MMLU's ceiling effect has made it one of the most widely used LLM evaluation tools since its introduction. Its acceptance as a Spotlight paper at NeurIPS 2024 further cemented its standing in the research community.^[8]

See Also

• Language Model Evaluation
• Chain-of-Thought Prompting
• Multi-task Learning
• Knowledge Benchmarks
• Prompt Engineering

References

1. Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., & Chen, W. (2024). "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark." *Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), Datasets and Benchmarks Track*. arXiv:2406.01574 ↩
2. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). "Measuring Massive Multitask Language Understanding." *Proceedings of the International Conference on Learning Representations (ICLR 2021)*. arXiv:2009.03300
3. TIGER-AI Lab. "MMLU-Pro GitHub Repository." https://github.com/TIGER-AI-Lab/MMLU-Pro
4. TIGER-AI Lab. "MMLU-Pro Dataset on Hugging Face." https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro ↩
5. TIGER-AI Lab. "MMLU-Pro Leaderboard." https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro ↩
6. Artificial Analysis. "MMLU-Pro Benchmark Leaderboard." https://artificialanalysis.ai/evaluations/mmlu-pro ↩
7. Vals AI. "MMLU Pro Benchmark." https://www.vals.ai/benchmarks/mmlu_pro ↩
8. NeurIPS 2024. "Poster: MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark." https://neurips.cc/virtual/2024/poster/97435 ↩