MMMU

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark) is a multimodal AI benchmark of 11,550 college-level questions that pairs text with images to test expert knowledge and reasoning across 30 academic subjects and 6 disciplines. It was released in November 2023 by a team of 22 researchers led by Xiang Yue (Ohio State University) and Wenhu Chen (University of Waterloo), and the paper states that MMMU is "designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning."^[1] As of June 21, 2026, the highest reported MMMU score is 86.0% (Qwen3.6 Plus), just inside the human expert range of 76.2% to 88.6%, so the benchmark is not yet saturated.^[1][8]

MMMU
Overview
Full name	Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark
Abbreviation	MMMU
Description	A massive multi-discipline multimodal benchmark evaluating expert-level understanding and reasoning across college-level subjects
Release date	2023-11
Latest version	1.0
Benchmark updated	2023-12-04
Authors	Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, et al. (22 authors)
Organizations	Ohio State University, University of Waterloo, IN.AI Research, Carnegie Mellon University
Venue	CVPR 2024 (Oral)
Technical Details
Type	Multimodal Understanding, Expert Knowledge
Modality	Text, Vision (Images)
Task format	Multiple choice (94%), Open-ended (6%)
Number of questions	11,550
Data splits	Dev: 150, Validation: 900, Test: 10,500
Subjects	30 subjects across 183 subfields
Image types	30+ heterogeneous types
Evaluation metric	Accuracy, Zero-shot performance
Domains	Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering
Languages	English
Performance
Human expert range	76.2% (worst) to 82.6% (medium) to 88.6% (best)
Random guess baseline	22.1% (validation)
SOTA score	86.0%
SOTA model	Qwen3.6 Plus
SOTA date	2026-06
Saturated	No
Resources
Website	Official website
Paper	arXiv:2311.16502
GitHub	Repository
Dataset	Hugging Face
Evaluation server	EvalAI
Successors	MMMU-Pro, Video-MMMU, CMMMU

MMMU contains 11,550 meticulously collected questions sourced from college exams, quizzes, and textbooks.^[1] It spans 30 subjects, 183 subfields, and six core academic disciplines, and features over 30 different image types ranging from standard photographs to specialized notations like chemical structures and music sheets.^[1] The benchmark was presented as an oral paper at CVPR 2024 and has since become one of the most widely used evaluations for assessing artificial general intelligence capabilities in multimodal contexts.^[1]

What is MMMU used for?

MMMU addresses a critical gap in AI evaluation by testing models on tasks that require both advanced visual perception and domain-specific knowledge reasoning. Unlike earlier multimodal benchmarks that focused on elementary visual understanding (identifying objects in photos, reading text from signs, or answering simple questions about natural images), MMMU demands college-level subject expertise combined with sophisticated reasoning about diverse visual content.^[1] The questions mirror the kind of problems students encounter in university courses across the sciences, humanities, engineering, and professional fields.

The benchmark is named for its defining characteristics: "Massive" refers to the scale of 11,550 questions; "Multi-discipline" indicates coverage across six broad academic disciplines; "Multimodal" highlights that questions combine text and images; and "Understanding and Reasoning" reflects the higher-order cognitive skills required. The full title is the Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, signaling the authors' goal of measuring progress toward human-expert-level artificial general intelligence.^[1] As the paper puts it, the authors hope "MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence."^[1]

Why was MMMU created?

The development of MMMU was motivated by several observations about the state of multimodal AI evaluation in 2023:

• Existing multimodal benchmarks primarily tested basic visual recognition (such as VQA, OK-VQA, and TextVQA) rather than requiring genuine expert knowledge. Models could often score well on these benchmarks without deep understanding of the visual content.
• Real-world professional tasks in medicine, engineering, finance, and other fields require interpreting complex visual information with domain-specific expertise. No existing benchmark captured this intersection well.
• Progress toward artificial general intelligence requires mastery of diverse academic disciplines, not just language fluency or basic pattern recognition. A benchmark was needed to track whether multimodal systems could approach the performance of educated human experts.
• Current models showed particularly poor generalization to less common visual formats like circuit diagrams, chemical structures, and musical notation, but no standardized evaluation existed to quantify this weakness.

The authors explicitly framed MMMU as a tool for the research community to measure progress toward "expert AGI," arguing that the ability to reason across multiple academic disciplines using varied visual inputs represents a meaningful milestone on the path to general intelligence.^[1]

How was the MMMU dataset built?

Collection Methodology

MMMU was built through a large-scale, human-driven collection effort. Over 50 college students from diverse academic backgrounds participated in gathering questions from textbooks, online educational resources, college exams, and lecture materials.^[1] The collection process followed strict guidelines to ensure quality, diversity, and difficulty.

Each question in the dataset was required to include at least one image that is essential to answering the question correctly. This design choice ensures that models cannot simply rely on text-based reasoning to bypass the visual component.^[1] The team collected questions from a wide range of sources, including university-level textbooks published by major academic publishers, past examination papers from accredited institutions, online educational platforms, and lecture slides from college courses.

Quality Control Pipeline

The dataset underwent a multi-stage review process:

1. Duplicate detection: Automated tools and manual review were used to identify and remove duplicate or near-duplicate questions across the entire collection.
2. Format standardization: All questions were converted to a consistent format with clear question text, answer options (for multiple-choice), and associated images.
3. Difficulty assessment: Questions were categorized into difficulty levels. Approximately 10% of initially collected questions were excluded for being too easy, as the benchmark targets expert-level reasoning.
4. Copyright compliance: The team ensured that all included content respected intellectual property guidelines.
5. Contamination prevention: Questions with widely available or easily searchable answers were flagged and reviewed to reduce the risk of models having memorized answers during pre-training.

Data Splits

The 11,550 questions are divided into three splits:

Scroll sideways for more →

The development set contains 5 questions per subject (150 total across 30 subjects). The validation set contains 30 questions per subject (900 total). The test set holds the remaining 10,500 questions, and its answer labels were kept private until February 2026, when the test set answers were publicly released.^[1]

Human Expert Baseline

To establish a meaningful human performance baseline, the MMMU team recruited 90 college senior students, with 3 experts assigned to each of the 30 subjects. Each expert completed 30 questions from the validation set within their discipline. Experts were allowed to consult their textbooks but were prohibited from searching the Internet for answers. Human expert accuracy on the validation set was reported at three levels: 76.2% (worst expert), 82.6% (medium expert), and 88.6% (best expert), against a random-guess baseline of 22.1%. This range provides a target for AI models to match or exceed.^[1]

Technical Specifications

Discipline and Subject Coverage

MMMU covers six core academic disciplines, each containing multiple subjects:^[1]

Scroll sideways for more →

The 30 subjects are further broken down into 183 subfields. For example, within Physics, subfields include classical mechanics, thermodynamics, electromagnetism, optics, and quantum physics. This granularity ensures that the benchmark captures a wide spectrum of college-level knowledge.

Image Type Diversity

One of MMMU's distinguishing features is its inclusion of over 30 heterogeneous image types. Most prior benchmarks focused on natural photographs, but MMMU deliberately includes many specialized visual formats that professionals and students encounter in their fields:^[1]

Scroll sideways for more →

Each question can include up to seven images, allowing the benchmark to test reasoning about complex multi-image scenarios such as comparing two X-rays or analyzing a series of related diagrams.^[1]

Question Types and Formats

Scroll sideways for more →

The heavy emphasis on multiple-choice questions allows for automated and unambiguous evaluation. Open-ended questions are included to test whether models can generate correct answers without the benefit of answer choices.

Evaluation Methodology

Zero-shot Protocol

MMMU employs strict zero-shot evaluation:

• No fine-tuning on MMMU data is allowed.
• No few-shot examples are provided during testing.
• Models must rely entirely on capabilities acquired during pre-training and general instruction tuning.
• This protocol ensures fair comparison across different model families and training approaches.

Skill Dimensions

The benchmark is designed to evaluate three core skill dimensions:

Scroll sideways for more →

Many MMMU questions require all three skills simultaneously. For instance, a question about organic chemistry might require recognizing a molecular structure (perception), knowing reaction mechanisms (knowledge), and predicting the product of a specific reaction (reasoning).

Difficulty Levels

Questions in MMMU are categorized by difficulty:

Scroll sideways for more →

The sharp drop-off from Easy to Hard questions illustrates that even advanced models struggle significantly once genuine expert-level reasoning is required.

How well do AI models score on MMMU?

Original Paper Results (November 2023)

When MMMU was first released, the authors evaluated a range of proprietary and open-source large multimodal models. The results revealed a substantial gap between the best models and human experts. The paper noted that "even the advanced GPT-4V only achieves a 56% accuracy, indicating significant room for improvement."^[1]

Scroll sideways for more →

Key findings from the original evaluation:

1. Even the strongest proprietary models fell well short of average human expert performance.
2. Open-source models (ranging from 13B to 34B parameters) scored roughly 20 percentage points below GPT-4V, averaging around 34% accuracy.
3. Models performed best in Humanities and Art & Design, where visual complexity tends to be lower and language-based reasoning carries more weight.
4. All models struggled significantly with uncommon image types such as geometric shapes, music sheets, and chemical structures, sometimes performing near random chance.

What is the current MMMU state of the art?

Since its release, MMMU has been widely adopted as a standard evaluation benchmark. Top model scores have improved substantially, with the best systems now exceeding 85% accuracy. As of June 21, 2026, the highest reported MMMU score on the LLM-Stats leaderboard is 86.0%, held by Alibaba's Qwen3.6 Plus, which narrowly edges out the GPT-5.1 family at 85.4%. The following table shows a selection of notable scores from the current leaderboard:^[8]

Scroll sideways for more →

Several models now surpass the lower end of human expert performance (76.2%), but the best human experts (88.6%) still outperform all current systems. The top-performing model, Qwen3.6 Plus at 86.0%, falls within the human expert range of 76.2% to 88.6% and is approaching the upper bound, which is why MMMU is still listed as unsaturated.^[8]

Performance by Discipline

Models consistently show uneven performance across the six disciplines. Humanities and Social Science questions tend to yield the highest scores, while Tech and Engineering questions remain the most challenging:

Scroll sideways for more →

Performance by Image Type

The type of visual content in a question has a major impact on model accuracy. Models trained primarily on natural images and web content tend to struggle with specialized visual formats:

Scroll sideways for more →

Error Analysis

The MMMU authors and subsequent researchers have identified four primary categories of model errors:^[1]

Scroll sideways for more →

These error categories are not mutually exclusive. A single incorrect answer may involve both a perception error (misreading part of an image) and a reasoning failure (drawing an incorrect conclusion from the misread data).

MMMU-Pro

Overview

MMMU-Pro is a more challenging successor benchmark introduced in September 2024 by a largely overlapping team of researchers, including Xiang Yue, Tianyu Zheng, Yuansheng Ni, and others. The paper was accepted at ACL 2025.^[2] MMMU-Pro was designed to address limitations in the original MMMU by filtering out questions that could be solved through shortcuts and introducing harder evaluation conditions.^[2] The authors report that across models, accuracy fell from 16.8% to 26.9% relative to MMMU once these harder conditions were applied.^[2]

Three-Step Construction Process

MMMU-Pro was constructed from the original MMMU dataset through a rigorous three-step process:^[2]

Step 1: Text-only filtering. The team used four strong open-source LLMs (Llama3-70B-Instruct, Qwen2-72B-Instruct, Yi-1.5-34B-Chat, and Mixtral-8x22B-Instruct) to identify questions that could be answered correctly without seeing the image. Each model attempted each question text-only across ten trials. Questions that were answered correctly by at least three of the four models more than five times were excluded. This process ensured that every remaining question genuinely requires visual understanding.^[2]

Step 2: Augmenting candidate options. For the remaining questions, human experts working alongside GPT-4o expanded the multiple-choice options from 4 to 10. This makes random guessing far less effective (10% chance versus 25%) and forces models to discriminate among more plausible distractors. During this phase, 70 additional questions were removed because the image-question relevance was insufficient, leaving 1,730 standard-format questions.^[2]

Step 3: Vision-only input setting. Human annotators manually captured screenshots and photographs of the questions displayed on screens, with varying backgrounds, font styles, and font sizes. This created a parallel set of 1,730 "vision-only" questions where the model must extract the question text from the image itself, testing integrated visual and textual processing without separate text input.^[2]

The final MMMU-Pro dataset contains 3,460 questions (1,730 standard + 1,730 vision-only), evenly distributed across the same 30 subjects as the original MMMU (approximately 60 questions per subject before vision-only duplication).^[2]

MMMU-Pro Results

Performance on MMMU-Pro is dramatically lower than on the original MMMU. The following table shows results from the original MMMU-Pro paper:^[2]

Scroll sideways for more →

The sharp performance drops demonstrate that a significant portion of MMMU accuracy came from shortcuts and guessing strategies rather than genuine multimodal understanding.

MMMU-Pro Key Findings

• Option augmentation is highly effective. Expanding from 4 to 10 choices reduced GPT-4o's accuracy by 10.7 percentage points even before applying the vision-only setting, showing that models relied partly on process-of-elimination strategies.^[2]
• Vision-only input is challenging. Embedding questions within images caused additional performance drops, particularly for models like LLaVA-OneVision-72B (14.0 percentage point decrease), indicating that many models struggle with integrated text-image processing.^[2]
• OCR prompts have minimal effect. Explicitly prompting models to perform OCR on the vision-only inputs did not significantly improve performance, suggesting that capable models already extract text effectively but struggle with the deeper reasoning challenges.^[2]
• Chain of Thought helps selectively. Chain of Thought prompting improved some models substantially (Claude 3.5 Sonnet rose from 42.7% to 55.0% in the standard setting) but hurt others, particularly smaller models with weaker instruction-following abilities.^[2]

Current MMMU-Pro Leaderboard (June 2026)

As models have improved, MMMU-Pro scores have risen considerably from the original paper's results. As of June 21, 2026, the LLM-Stats MMMU-Pro leaderboard is led by Google's Gemini 3.5 Flash at 83.6%, ahead of OpenAI's GPT-5.5 at 83.2%:^[9]

Scroll sideways for more →

MMMU Family of Benchmarks

The success of MMMU has led to the development of several related benchmarks, forming a broader "MMMU family" that evaluates different aspects of multimodal understanding:

CMMMU (Chinese MMMU)

CMMMU (Chinese Massive Multi-discipline Multimodal Understanding) was released in early 2024 as a Chinese-language counterpart to MMMU. It contains approximately 12,000 manually collected multimodal questions covering the same six disciplines and 30 subjects as MMMU, but sourced from Chinese educational curricula. CMMMU includes 39 heterogeneous image types and tests models on Chinese-specific academic content. Even GPT-4V only achieved approximately 42% accuracy on CMMMU, highlighting the additional challenge of non-English academic evaluation.^[4]

Video-MMMU

Video-MMMU extends the MMMU paradigm to video understanding. Developed by researchers at Nanyang Technological University and Carnegie Mellon University, Video-MMMU contains 300 expert-level, college-level lecture videos and 900 human-annotated questions across the same six disciplines and 30 subjects.^[3] The benchmark evaluates knowledge acquisition through three cognitive stages: Perception (identifying key information), Comprehension (understanding underlying concepts), and Adaptation (applying knowledge to novel scenarios). A novel metric called delta-knowledge measures how much a model's performance improves after watching an educational video. Human learners achieved a 33.1% knowledge gain, while GPT-4o achieved only 15.6% and Claude 3.5 Sonnet achieved 11.4%, revealing a significant gap in video-based learning capabilities.^[3]

Uni-MMMU

Uni-MMMU, published in late 2025, is a unified benchmark that tests bidirectional synergy between generation and understanding across eight reasoning-centric domains including science, coding, mathematics, and puzzles. Each task is bidirectionally coupled, requiring models either to leverage conceptual understanding to guide precise visual synthesis or to use generation as a cognitive scaffold for analytical reasoning.

Applications and Impact

Significance in AI Research

MMMU has become one of the most widely cited and used benchmarks for multimodal AI evaluation since its release. Its significance stems from several factors:

• It was the first large-scale benchmark to systematically test multimodal models on college-level expert knowledge across multiple disciplines.
• Major AI companies including OpenAI, Google, Anthropic, and Meta routinely report MMMU scores when releasing new multimodal models.
• The benchmark has helped identify specific weaknesses in vision-language models, such as poor handling of specialized visual formats and weak cross-modal reasoning.
• MMMU's design philosophy, prioritizing expert-level reasoning over basic visual understanding, has influenced the development of other challenging benchmarks.

Research Applications

MMMU enables several lines of research:

• Multimodal architecture design: Identifying which model architectures best handle diverse visual inputs combined with knowledge-intensive reasoning.
• Knowledge integration: Studying how to combine visual perception with domain expertise during both pre-training and fine-tuning.
• Visual generalization: Understanding why models transfer poorly to uncommon image formats and developing training strategies to address this.
• Curriculum learning: Using MMMU's difficulty levels to design progressive training regimes.
• Error analysis: Diagnosing whether model failures stem from perception, knowledge, or reasoning limitations.

Professional and Educational Applications

Scroll sideways for more →

Dataset Access and Usage

Access Methods

MMMU is publicly available through Hugging Face Datasets:^[6]

from datasets import load_dataset

# Load the full MMMU dataset
dataset = load_dataset("MMMU/MMMU")

# Access specific subjects
accounting = load_dataset("MMMU/MMMU", "Accounting")
physics = load_dataset("MMMU/MMMU", "Physics")

The dataset is also available for direct download from the Hugging Face repository.^[6]

Evaluation Server

• Launch date: December 4, 2023
• Platform: EvalAI
• URL: https://eval.ai/web/challenges/challenge-page/2179/leaderboard
• Submission format: JSON file with model predictions
• Test set answers released: February 12, 2026 (local evaluation now possible)

Each Data Sample

Every question in MMMU includes the following fields:

Scroll sideways for more →

Limitations

Scroll sideways for more →

Future Directions

Several research directions build on the foundation laid by MMMU:

1. Multilingual expansion: Extending the benchmark to additional languages beyond English and Chinese. CMMMU covers Chinese, but benchmarks for other major languages remain needed.
2. Dynamic evaluation: Developing procedurally generated or frequently updated question sets to prevent contamination and memorization.
3. Interactive evaluation: Multi-turn reasoning tasks where models can ask clarifying questions or request additional information.
4. Video and temporal reasoning: Extending to video-based questions requiring understanding of temporal sequences and dynamic visual content.
5. Fine-grained skill assessment: More detailed breakdowns of perception, knowledge, and reasoning sub-skills to identify specific model weaknesses.
6. Unified generation and understanding: Benchmarks like Uni-MMMU that test whether models can both generate and understand visual content across disciplines.

Related Benchmarks

• MMLU: Text-only multi-task language understanding benchmark covering 57 subjects
• MMLU-Pro: Enhanced version of MMLU with harder questions and 10 answer options
• MathVista: Mathematical reasoning with visual inputs
• ScienceQA: Multimodal science questions from elementary to high school level
• ChartQA: Chart and data visualization understanding
• AI2D: Science diagram understanding
• TextVQA: Text reading in natural images
• DocVQA: Document understanding and visual question answering
• HumanEval: Code generation benchmark (text-only, for comparison)
• VQA v2.0: General visual question answering on natural images

See Also

• MM-BrowseComp
• MMStar
• Multimodal AI
• Vision-Language Models
• Computer Vision
• Artificial General Intelligence
• Large Language Models
• MMLU
• GPT-4
• Gemini

References

1. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., & Chen, W. (2024). "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024)*. https://arxiv.org/abs/2311.16502 ↩
2. Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S., Sun, Y., Yu, B., Zhang, G., Sun, H., Su, Y., Chen, W., & Neubig, G. (2025). "MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark." *Proceedings of ACL 2025*. https://arxiv.org/abs/2409.02813 ↩
3. Hu, K., Wu, P., Pu, F., Xiao, W., Yue, X., Zhang, Y., Li, B., & Liu, Z. (2025). "Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos." https://videommmu.github.io/ ↩
4. Zhang, G., Du, Y., et al. (2024). "CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark." https://arxiv.org/abs/2401.11944 ↩
5. MMMU Benchmark Official Website. https://mmmu-benchmark.github.io/
6. MMMU Dataset on Hugging Face. https://huggingface.co/datasets/MMMU/MMMU ↩
7. MMMU GitHub Repository. https://github.com/MMMU-Benchmark/MMMU
8. MMMU Leaderboard, LLM-Stats (accessed June 21, 2026). https://llm-stats.com/benchmarks/mmmu ↩
9. MMMU-Pro Leaderboard, LLM-Stats (accessed June 21, 2026). https://llm-stats.com/benchmarks/mmmu-pro ↩