MLPerf

MLPerf is the AI industry's standard suite of benchmarks for measuring how fast machine learning systems train and run models, developed and maintained by MLCommons, a nonprofit engineering consortium of more than 125 organizations. Founded in 2018, MLPerf measures the wall-clock performance of GPU and accelerator hardware on real-world workloads such as large language model pretraining, image generation, and recommendation, and it has become the most widely cited benchmark in the AI hardware industry. By 2026 the suite had produced more than 100,000 individual results, and its headline training record stood at 27.3 minutes to pretrain Llama 3.1 405B on 2,496 NVIDIA GB200 GPUs. Chip manufacturers, cloud providers, and system integrators use MLPerf results to validate performance claims and guide multimillion-dollar purchasing decisions.^[1][5][8]

MLCommons states that MLPerf exists to "accelerate ML progress through fair and useful measurement," enabling fair comparison of competing systems while still encouraging AI innovation, enforcing reproducibility, serving both commercial and research communities, and keeping benchmarking affordable enough that all can participate.^[6]

History and Origins

The MLPerf project originated in February 2018 from a series of meetings between engineers and researchers at Baidu, Google, Harvard University, Stanford University, and UC Berkeley. The founding team included Peter Mattson (Google Brain), David Patterson (UC Berkeley), Greg Diamos (Baidu), Cliff Young (Google), Peter Bailis (Stanford), and Gu-Yeon Wei (Harvard). Their goal was to create an equivalent of SPEC benchmarks or TPC benchmarks for the machine learning field, providing standardized, fair comparisons across rapidly proliferating hardware and software platforms.^[1]

The first MLPerf Training benchmark results were published in late 2018, and the MLPerf Inference benchmark followed in 2019. The formal academic paper, "MLPerf Training Benchmark," appeared on arXiv in October 2019 with contributions from 37 authors spanning industry and academia.^[1]

In December 2020, the MLPerf effort was formally incorporated into MLCommons, a nonprofit engineering consortium headquartered in Dover, Delaware. MLCommons expanded the scope beyond benchmarking to include open datasets, safety evaluations, and research initiatives. The founding board included representatives from Alibaba, Meta (then Facebook AI), Google, Intel, NVIDIA, and Harvard University.^[5]

As of 2026, MLCommons has grown to include over 125 member organizations ranging from startups to Fortune 500 companies, universities, and nonprofits across the globe. The MLPerf benchmark suite has produced more than 100,000 individual results across all its categories.^[5][7]

Organizational Structure

MLCommons organizes its work through several working groups:

Scroll sideways for more →

The MLPerf benchmarks follow a biannual release cadence, with new results published roughly every six months (e.g., v5.0 in June 2025 and v5.1 in November 2025 for Training; v5.1 in September 2025 and v6.0 in April 2026 for Inference).^[7][8][9] Each release can include updated workloads, new models, and retired benchmarks to keep pace with the fast-moving field.

What does the MLPerf benchmark suite include?

MLPerf is not a single benchmark but a family of benchmark suites, each targeting a different aspect of the ML lifecycle or a different class of hardware. The major suites are:

Scroll sideways for more →

Submission Divisions and Categories

All MLPerf benchmark suites use a common framework of submission divisions and availability categories to ensure fair comparisons while still allowing innovation.

Divisions

Closed Division: Submitters must use the reference model architecture and adhere to strict rules about preprocessing, training procedures, and hyperparameters. This division enables direct, apples-to-apples comparisons of hardware and system software. Most industry submissions fall into this category.^[1][2]

Open Division: Submitters may modify the model architecture, use different optimizers, or apply novel training techniques. This division encourages algorithmic innovation and research exploration. Results in the Open Division cannot be directly compared to Closed Division results.^[1][2]

Availability Categories

Scroll sideways for more →

MLPerf Training

The MLPerf Training benchmark measures the wall-clock time required to train a model on a specified dataset to a defined quality target. The methodology involves running the benchmark a specified number of times, discarding the highest and lowest results, and averaging the remainder. Typical accuracy variance across runs is roughly plus or minus 2.5% for imaging benchmarks and plus or minus 5% for other workloads.^[1]

Training Workloads (v5.1, November 2025)

The v5.1 suite includes the following workloads:^[9]

Scroll sideways for more →

Notable Training Results

MLPerf Training v5.0 (June 2025) introduced the Llama 3.1 405B benchmark, the largest model ever included in the training suite. In a landmark result, CoreWeave, NVIDIA, and IBM completed the Llama 3.1 405B training benchmark in just 27.3 minutes using a cluster of 2,496 NVIDIA GB200 NVL72 GPUs. This was the largest cluster ever benchmarked under MLPerf and achieved 91% scaling efficiency from 512 to 2,496 GPUs.^[8][16]

The NVIDIA Blackwell architecture delivered up to 2.6x higher performance per GPU compared to the previous-generation Hopper architecture, with at least 2x faster training at equivalent cluster sizes.^[15]

MLPerf Training v5.1 (November 2025) included results from 20 organizations submitting 65 unique systems across 12 different hardware accelerators. Nearly half of all submissions were multi-node configurations, an 86% increase over v4.1. The generative AI workloads (Llama 2 70B LoRA and Llama 3.1 8B) saw 24% and 15% submission increases respectively.^[9] First-time submitters included University of Florida, Verda, and Wiwynn.^[9]

NVIDIA swept all seven MLPerf Training v5.1 tests and was the only platform to submit results on every test.^[17] Highlights included a 10-minute Llama 3.1 405B pretraining time on 5,000+ Blackwell GPUs, a 5.2-minute Llama 3.1 8B pretraining time on 512 Blackwell Ultra GPUs, and a 12.5-minute Flux.1 image-generation training on 1,152 Blackwell GPUs.^[17] Compared to Hopper, Blackwell Ultra (GB300 NVL72) delivered roughly 4x faster Llama 3.1 405B pretraining and nearly 5x faster Llama 2 70B LoRA fine-tuning at the same GPU count.^[17] NVIDIA described NVFP4 as "a first in the history of MLPerf Training," noting it was "the only company that had submitted MLPerf Training results for FP4 precision while simultaneously meeting the strict accuracy requirements of the benchmark."^[17]

NVIDIA also demonstrated the first MLPerf training submissions using NVFP4 four-bit floating-point precision, achieving double the rate of FP8 on Blackwell and roughly 3x on Blackwell Ultra while remaining the only platform meeting accuracy requirements with FP4.^[17]

Version 4.0 (2024) was notable for introducing system-wide power draw and energy consumption measurements during training, reflecting the growing importance of energy efficiency in large-scale AI systems.

MLPerf Inference

The MLPerf Inference benchmark measures how quickly systems can execute trained models across various deployment scenarios. It covers both datacenter and edge deployments, with different scenarios reflecting real-world usage patterns.^[2]

Inference Scenarios

Scroll sideways for more →

Datacenter submissions require the Offline and Server scenarios (plus Interactive for LLM benchmarks). Edge submissions require SingleStream, MultiStream, and Offline scenarios.^[2][7]

Inference Workloads (v6.0, April 2026)

MLPerf Inference v6.0, released April 1, 2026, refreshed roughly half of the datacenter suite. Five of the eleven datacenter tests were new or updated, and an additional new object-detection workload was added for edge systems.^[7]

Scroll sideways for more →

LoadGen++ and Visualizer

The v6.0 release also introduced LoadGen++, a substantially upgraded version of the load-generator harness that allows LLM benchmarks to run against serving-style software stacks (resembling production deployments) rather than the simpler bench rigs used previously. MLCommons described LoadGen++ as a key infrastructure investment to keep the suite nimble as model serving stacks evolve.^[7] Alongside LoadGen++, MLCommons launched an interactive results dashboard at mlcommons.org/visualizer with advanced filtering and customized performance graphs.^[7][20]

Accuracy Requirements

Most benchmarks require the submitted system to achieve at least 99% of the accuracy of the FP32 reference model. Language processing and medical imaging benchmarks have a stricter threshold of 99.9% accuracy relative to the reference. For GPT-OSS 120B, accuracy targets are 82.92% on AIME (8 repeats), 74.95% on GPQA-Diamond (5 repeats), and 84.68% on LiveCodeBench v6 (3 repeats).^[19]

Notable Inference Results

MLPerf Inference v6.0 (April 2026) drew submissions from 24 organizations, including AMD, ASUSTeK, Cisco, CoreWeave, Dell, GATEOverflow, Giga Computing, Google, HPE, Intel, Inventec, KRAI, Lambda, Lenovo, MangoBoost, MiTAC, Nebius, Netweb Technologies India, NVIDIA, Oracle, Quanta Cloud Technology, Red Hat, Stevens Institute of Technology, and Supermicro.^[7] The round saw a 30% increase in multi-node submissions versus v5.1, and 10% of submissions featured more than ten nodes.^[7]

NVIDIA's largest submission used 288 Blackwell Ultra GPUs across four GB300 NVL72 racks connected by Quantum-X800 InfiniBand, the largest MLPerf Inference submission ever, quadruple the previous round's maximum.^[7][17][20] On DeepSeek-R1, this system delivered approximately 2.49 million tokens/second offline and 1.55 million tokens/second in server mode.^[20] Blackwell Ultra (GB300) also set per-GPU records on GPT-OSS 120B, Llama 3.1 405B, Llama 3.1 8B, Whisper, and other workloads.^[17][20]

AMD's Instinct MI355X delivered competitive single-node results and the first model bring-up on GPT-OSS 120B and Wan 2.2 text-to-video by a non-NVIDIA accelerator, and AMD multinode submissions crossed one million tokens per second at cluster scale.^[7] Intel was the only vendor with standalone CPU submissions; over half of the v6.0 submissions used Xeon processors, and Intel debuted its Arc Pro B70 GPU (delivering up to 1.8x the inference performance of the B60).^[18]

MLPerf Inference v5.1 (September 2025) had set the previous participation record at 27 organizations. Five new accelerators debuted: AMD Instinct MI355X, Intel Arc Pro B60 48GB Turbo, NVIDIA GB300, NVIDIA RTX 4000 Ada PCIe 20GB, and NVIDIA RTX Pro 6000 Blackwell Server Edition. Performance on Llama 2 70B improved by up to 50% over v5.0 results from just six months prior. The round also included the first heterogeneous system submission, which used software load-balancing across different accelerator types.

MLPerf Inference v5.0 (April 2025) included 17,457 results from 23 organizations. Five newly available processors were benchmarked: AMD Instinct MI325X, Intel Xeon 6980P, Google TPU Trillium (v6e), NVIDIA B200, and NVIDIA Jetson AGX Thor 128.

MLPerf HPC

MLPerf HPC targets supercomputer-scale systems used for scientific machine learning. Unlike the standard Training benchmark, HPC benchmarks focus on scientific applications and include an optional throughput metric for large multi-user systems.^[4]

HPC Workloads (v3.0, November 2023)

Scroll sideways for more →

MLPerf HPC v3.0 results demonstrated performance gains of up to 2.8x compared to just five months prior and 49x over the first HPC results. The DeepCAM weather modeling benchmark ran 14x faster than when it debuted, illustrating how innovations in ML hardware and software benefit scientific computing.^[10]

MLPerf Tiny

MLPerf Tiny benchmarks inference performance on extremely low-power embedded devices such as microcontrollers, DSPs, and tiny neural network accelerators. These devices typically operate at clock speeds between 10 MHz and 250 MHz and consume less than 50 mW of power. The neural networks tested are small, typically 100 KB and below, processing sensor data like audio and images.^[3]

Tiny Workloads (v1.3, September 2025)

Scroll sideways for more →

The reference implementation uses TensorFlow Lite for Microcontrollers (TFLM). Three metrics are measured for each benchmark: accuracy, latency, and energy consumption. MLPerf Tiny v1.3 included 70 results across five benchmarks from four participants: Kai Jiang, Qualcomm, STMicroelectronics, and Syntiant.^[12]

MLPerf Storage

MLPerf Storage measures how quickly storage systems can feed training data to accelerators during model training. Storage throughput is a critical bottleneck in large-scale training, especially when datasets are too large to fit in memory.

Storage Workloads

Scroll sideways for more →

The v2.0 results showed that submitted storage systems could support roughly twice the number of simultaneous accelerators compared to v1.0, reflecting the scaling demands of modern training clusters.^[11]

MLPerf Client

MLPerf Client evaluates AI performance on personal computers, including laptops, desktops, and workstations. Announced in January 2024, this benchmark suite targets the growing "AI PC" market segment where large language models run locally on consumer hardware.

Client Version History

Scroll sideways for more →

MLPerf Client v1.5 supports Windows x64, Windows on Arm, macOS, Linux, and iPad, making it the most cross-platform MLPerf benchmark. It is developed collaboratively by AMD, Intel, Microsoft, NVIDIA, Qualcomm, and major PC OEMs.^[13]

MLPerf Automotive

MLPerf Automotive is jointly developed by MLCommons and the Autonomous Vehicle Computing Consortium (AVCC) to evaluate inference performance for automotive AI systems. The benchmark covers Advanced Driver Assistance Systems (ADAS), autonomous driving (AD), and in-vehicle infotainment (IVI).^[14]

Automotive Workloads (v0.5, August 2025)

Scroll sideways for more →

The 3D object detection benchmark uses PointPainting, a sensor-fusion technique that combines image-based semantic segmentation with lidar point cloud data. Future versions are expected to incorporate vision-language-action (VLA) models for end-to-end self-driving evaluation.^[14]

AILuminate

AILuminate is the MLCommons safety and reliability benchmark for large language models and AI chat systems, run by the AI Risk and Reliability working group. First launched as v1.0 in December 2024 and updated to v1.1 in 2025, it is described by MLCommons as the first industry-standard benchmark for assessing AI-product risk.^[21][22]

AILuminate evaluates a system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior across twelve hazard categories: violent crimes, nonviolent crimes, sex-related crimes, child sexual exploitation, indiscriminate weapons (CBRNE), suicide and self-harm, intellectual property, privacy, defamation, hate, sexual content, and specialized advice (medical, legal, financial).^[21] Each test uses over 24,000 prompts per language (12,000 public practice prompts plus 12,000 private prompts used for the Official Test) and assigns systems a letter-grade safety score from "Poor" to "Excellent."^[21]

The benchmark initially supported English (v1.0) and added French, with Chinese and Hindi in development.^[21] AILuminate complements the performance-focused MLPerf suites by giving model developers and procurement teams a comparable measure of model safety alongside throughput and latency.

How do companies use MLPerf?

MLPerf results serve several purposes across the AI industry:

Hardware Vendors: Chip makers like NVIDIA, AMD, Intel, and Google use MLPerf to demonstrate the performance advantages of new processor generations. NVIDIA has been a particularly dominant and consistent top performer, using MLPerf results prominently in product launches for its Hopper, Blackwell, and Blackwell Ultra GPU architectures.^[15][17]

Cloud Providers: Companies like CoreWeave, Oracle Cloud, Google Cloud, Microsoft Azure, Nebius, Lambda, and Vultr use MLPerf to validate their infrastructure offerings. Cloud-based submissions allow potential customers to see performance on rentable systems.^[7][8]

System Integrators: Server manufacturers like Dell, HPE, Lenovo, Supermicro, Cisco, ASUSTeK, Quanta Cloud Technology, Inventec, and Wiwynn submit results to showcase the performance of their configured systems, helping enterprise buyers compare turnkey solutions.^[7][9]

AI Startups and Chip Designers: Newer entrants like Cerebras, SambaNova, Groq, and MangoBoost have used MLPerf submissions (or plan to) to establish credibility and benchmark their novel architectures against established players.^[7]

Researchers: Academic institutions use MLPerf HPC and other suites to measure the ML capabilities of research clusters and supercomputers, informing procurement and system design decisions. Recent first-time academic submitters include the University of Florida, Stevens Institute of Technology, and Amitash Nanda.^[7][9]

How does MLPerf ensure fair comparisons?

MLPerf employs several mechanisms to ensure fair and meaningful comparisons:^[1][2]

• Reference implementations are provided for every benchmark, giving all submitters a common starting point.
• Strict rules in the Closed Division control model architecture, preprocessing, and training hyperparameters.
• Multiple runs are required, with outliers discarded and results averaged to reduce variance.
• Peer review allows submitters to review each other's results before publication.
• Results usage guidelines establish standards for how companies can cite MLPerf numbers in marketing materials.
• Power measurement (optional but increasingly adopted) captures energy consumption alongside performance.
• Speculative decoding rules introduced in v6.0 require EAGLE-style implementation and prohibit manipulating acceptance rates via continued pretraining or quantization.^[19]

The biannual cadence and evolving workload selection ensure that benchmarks remain relevant as the field progresses. Retired workloads (such as the Mini Go reinforcement learning benchmark, the original NMT machine translation task, GPT-3, GPT-J, and BERT) are replaced with newer, more representative models.

What are the criticisms of MLPerf?

While MLPerf is widely respected, some criticisms have been raised:

• Vendor dominance in submissions: Large companies with dedicated benchmarking teams can optimize extensively, potentially making results less representative of typical user experience.
• Benchmark selection lag: Despite regular updates, the fast pace of AI development means some benchmarks may not reflect the latest model architectures at any given time.
• Cost and complexity: Participating in MLPerf requires significant engineering effort and hardware investment, which can limit participation from smaller organizations and startups.
• Closed Division constraints: The strict rules, while enabling fair comparison, may not reflect the optimizations that users would apply in production deployments.
• Coverage of emerging workloads: Although v6.0 added text-to-video and vision-language workloads, areas like long-horizon agentic AI, audio-conversational systems, and end-to-end multimodal pipelines remain underrepresented, though MLCommons continuously evaluates new workload candidates.^[7]

Version History Summary

Scroll sideways for more →

See Also

• BERT
• Llama
• ResNet
• NVIDIA
• GPU Computing
• Stable Diffusion
• Whisper
• Mixtral
• Deep Learning
• GLUE Benchmark
• MLCommons

References

1. Mattson, P., et al. "MLPerf Training Benchmark." arXiv:1910.01500, October 2019. https://arxiv.org/abs/1910.01500 ↩
2. Reddi, V.J., et al. "MLPerf Inference Benchmark." arXiv:1911.02549, November 2019. https://arxiv.org/abs/1911.02549 ↩
3. Banbury, C., et al. "MLPerf Tiny Benchmark." arXiv:2106.07597, June 2021. https://arxiv.org/abs/2106.07597 ↩
4. Farrell, S., et al. "MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems." arXiv:2110.11466, October 2021. https://arxiv.org/abs/2110.11466 ↩
5. MLCommons. "About MLCommons." https://mlcommons.org/about-us/. ↩
6. MLCommons. "MLPerf Training Benchmark." https://mlcommons.org/benchmarks/training/. ↩
7. MLCommons. "MLCommons Releases New MLPerf Inference v6.0 Benchmark Results." April 1, 2026. https://mlcommons.org/2026/04/mlperf-inference-v6-0-results/ ↩
8. MLCommons. "New MLCommons MLPerf Training v5.0 Benchmark Results Reflect Rapid Growth and Evolution of the Field of AI." June 4, 2025. https://mlcommons.org/2025/06/mlperf-training-v5-0-results/ ↩
9. MLCommons. "MLCommons Releases MLPerf Training v5.1 Results." November 12, 2025. https://mlcommons.org/2025/11/training-v5-1-results/ ↩
10. MLCommons. "New MLPerf Training and HPC Benchmark Results Showcase 49X Performance Gains in 5 Years." November 8, 2023. https://mlcommons.org/2023/11/mlperf-training-v3-1-hpc-v3-0-results/ ↩
11. MLCommons. "New MLPerf Storage v2.0 Benchmark Results." August 2025. https://mlcommons.org/2025/08/mlperf-storage-v2-0-results/ ↩
12. MLCommons. "MLCommons New MLPerf Tiny v1.3 Benchmark Results Released." September 2025. https://mlcommons.org/2025/09/mlperf-tiny-v1-3-results/ ↩
13. MLCommons. "MLPerf Client v1.5 Advances AI PC Benchmarking with Windows ML Integration." November 2025. https://mlcommons.org/2025/11/mlperf-client-1-5-release/ ↩
14. MLCommons. "AVCC and MLCommons Release New MLPerf Automotive v0.5 Benchmark Results." August 2025. https://mlcommons.org/2025/08/mlperf-auto-v0-5-results/ ↩
15. NVIDIA. "NVIDIA Blackwell Delivers up to 2.6x Higher Performance in MLPerf Training v5.0." https://developer.nvidia.com/blog/nvidia-blackwell-delivers-up-to-2-6x-higher-performance-in-mlperf-training-v5-0/ ↩
16. CoreWeave. "CoreWeave, NVIDIA and IBM Submit Largest-Ever MLPerf Results on NVIDIA GB200 Grace Blackwell Superchips." 2025. https://investors.coreweave.com/news/ ↩
17. NVIDIA. "NVIDIA Wins Every MLPerf Training v5.1 Benchmark." November 2025. https://blogs.nvidia.com/blog/mlperf-training-benchmark-blackwell-ultra/ ↩
18. Intel. "Intel Delivers Open, Scalable AI Performance in MLPerf Inference v6.0." April 2026. https://newsroom.intel.com/artificial-intelligence/intel-delivers-ai-performance-mlperf-inference-v6-0 ↩
19. MLCommons. "A new GPT-OSS benchmark and DeepSeek R1 updates for latency-optimized reasoning." March 2026. https://mlcommons.org/2026/03/mlperf-inference-gpt-oss/ ↩
20. NVIDIA. "NVIDIA Blackwell Ultra Sets New Inference Records in MLPerf Debut." April 2026. https://developer.nvidia.com/blog/nvidia-blackwell-ultra-sets-new-inference-records-in-mlperf-debut/ ↩
21. Ghosh, S., et al. "AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons." arXiv:2503.05731, March 2025. https://arxiv.org/abs/2503.05731 ↩
22. MLCommons. "AILuminate." https://mlcommons.org/benchmarks/ailuminate/ ↩