KernelBench

Overview

KernelBench is an AI benchmark and open-source evaluation environment that measures how well large language models can write fast and correct GPU kernels. The benchmark presents a model with a reference implementation written in PyTorch and asks it to produce a custom kernel, typically in CUDA, that computes the same result but runs faster on the GPU. Generated kernels are checked for numerical correctness against the PyTorch reference and then timed to measure any speedup ^[1][2].

KernelBench was first presented in a Stanford Scaling Intelligence Lab blog post dated December 3, 2024 and then described in the February 2025 paper "KernelBench: Can LLMs Write Efficient GPU Kernels?" by Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini, a team at Stanford (the Scaling Intelligence Lab) and Princeton University, with Ouyang and Guo as equal first authors. The paper was posted to arXiv on February 14, 2025 (arXiv:2502.10517) and was accepted to the International Conference on Machine Learning (ICML) 2025 ^[1][2][3]. The benchmark comprises 250 carefully selected neural-network workloads, and it introduced a metric called fast_p that has since become a common way to report progress on automated GPU kernel generation ^[1][2].

The starting premise of KernelBench is that writing efficient GPU kernels is a high-value but difficult engineering task, and that progress on the benchmark translates directly into faster real-world machine-learning code rather than into an abstract score ^[2]. Through 2025 it became one of the standard reference points for evaluating "AI writing CUDA" efforts at companies and labs including NVIDIA ^[4].

Motivation (GPU kernel generation)

A GPU kernel is a small program that runs on a graphics processor and performs a specific computation, such as a matrix multiplication, a convolution, or a normalization step. Modern deep-learning models spend almost all of their compute inside such kernels, so the speed of these low-level routines largely determines how fast a model trains and how cheaply it runs at inference time ^[2].

Writing a fast kernel is hard. It requires detailed knowledge of GPU architecture, including the memory hierarchy, thread and block scheduling, shared memory, and how to keep the hardware's arithmetic units busy. It also tends to be tedious and error prone, and the people who can do it well are scarce. As a result, much of this work is done by a small number of specialists or is delegated to compilers and hand-tuned libraries ^[2].

The KernelBench authors frame this as a natural target for AI code generation. If a language model could reliably produce kernels that are both correct and faster than the default implementation, it would automate a costly bottleneck in machine-learning systems engineering. Unlike many coding benchmarks, the objective here is not only functional correctness but also measurable performance, and the reward signal (a real speedup on real hardware) is concrete and verifiable ^[1][2]. This makes KernelBench an example of an evaluation with an executable, performance-based ground truth rather than a fixed reference answer.

Structure (the levels)

KernelBench organizes its 250 core tasks into three difficulty levels, with an additional aspirational level drawn from real models. Each task provides a PyTorch reference inside a Model class with __init__ and forward methods, together with helper functions (get_inputs and get_init_inputs) that generate test tensors of fixed shapes. The model under test must return a replacement that produces the same outputs, usually by writing custom CUDA kernels in place of the PyTorch operations ^[1][2].

The levels are ordered by the number of primitive operations a task contains, not by measured score: in the paper's one-shot results Level 2 was in fact the easiest level for the leading reasoning models ^[1]. Level 1 isolates a single operation, where a model only has to optimize one kernel. Level 2 requires the model to recognize and exploit fusion opportunities, combining several operations into one kernel to cut memory traffic. Level 3 demands an optimized implementation of an entire architecture, which is closest to a realistic engineering task. The 20 Level 4 tasks, described by the authors as aspirational, target the optimization of complete models taken from a model hub and are harder still ^[1][2].

The benchmark deliberately does not impose a fixed train/test split, leaving researchers free to choose their own evaluation protocol, and it evaluates kernels only on fixed input shapes rather than requiring them to generalize across arbitrary shapes ^[1].

The fast_p metric

The headline metric of KernelBench is fast_p. It is defined as the fraction of tasks for which the generated kernel is both functionally correct and achieves a speedup greater than an adjustable threshold p over a baseline implementation. Speedup is computed as the ratio of the baseline wall-clock time to the generated kernel's wall-clock time, and only correct solutions can count toward the metric, since the authors note that fast but incorrect code is useless ^[1][2].

The threshold p makes the metric tunable, which lets a single benchmark express several questions at once ^[1][2]:

• fast_0 counts kernels that are merely correct, with no speed requirement (a threshold of zero). This is equivalent to the plain correctness or pass rate.
• fast_1 counts kernels that are correct and at least match the baseline, that is, run as fast or faster than PyTorch.
• fast_2 counts kernels that are correct and at least twice as fast as the baseline.

As p rises, the bar gets harder and scores fall, so reporting fast_p across several thresholds shows not just whether a model can produce working kernels but how aggressively it can optimize them ^[1][2]. Correctness itself is established empirically: the generated kernel is run on five sets of random inputs of the fixed shapes, and its outputs must match the reference within an absolute and relative tolerance of 1e-02. Baseline timings are taken against PyTorch, and the framework supports comparison against both standard ("eager") PyTorch and the torch.compile baseline. Because timing is sensitive to the specific GPU, drivers, and software versions, the authors recommend regenerating baseline times on the same hardware used for evaluation ^[6].

Results

At the time of release, KernelBench was difficult for frontier models. The authors reported that the strongest reasoning models performed best out of the box but still fell short overall, matching or beating the PyTorch baseline (fast_1) in fewer than 20 percent of tasks. In the paper's one-shot table, measured on an NVIDIA L40S GPU against PyTorch Eager, reasoning-focused models led at every level: OpenAI's o1 scored fast_1 of 10, 24 and 12 percent on Levels 1 to 3 and DeepSeek-R1 12, 36 and 2 percent, against 4, 5 and 0 percent for GPT-4o. The authors' failure analysis found that reasoning models produced fewer kernels with execution failures, but that all models struggled similarly with functional correctness ^[1][2].

The paper also showed that performance improves substantially when models are allowed to use execution and profiling feedback during iterative refinement, repeatedly running a candidate kernel, observing errors and timing data, and trying again. Even so, KernelBench remained a hard benchmark, and the authors stressed that its difficulty grows as the speedup threshold p is raised, since clearing fast_2 or higher requires genuine optimization rather than a correct but unremarkable kernel. They also highlighted a tension between the two objectives, because correctness and aggressive performance optimization are often in conflict ^[1][2].

A widely cited follow-on result came from NVIDIA. In a developer blog dated February 12, 2025, NVIDIA engineers described a closed-loop "inference-time scaling" workflow that paired the DeepSeek-R1 reasoning model with an automated verifier running on an H100 GPU. By letting the system generate, verify, and refine kernels in a loop, they reported numerically correct kernels for 100 percent of Level 1 problems and 96 percent of Level 2 problems in KernelBench, and noted that allocating more than 10 minutes of inference-time compute per Level 1 problem was enough to solve most of the 100 problems in that level, illustrating how additional test-time compute can close much of the gap that frontier models showed when answering in a single pass ^[4]. These figures measure correctness produced by an agentic system with extended search rather than the single-shot fast_1 rates reported in the original paper, so the two sets of numbers are not directly comparable.

Use in automated discovery systems

Because each KernelBench task comes with an executable correctness check and a measurable runtime, individual tasks have also been used as testbeds for automated discovery systems that search over many candidate kernels rather than answering once. One example is Dream-RSI, a framework from Google, Google DeepMind, the University of Maryland and the University of Virginia posted to arXiv on September 14, 2026, which improves the exploration policy of a discovery loop by replaying alternative policies over the recorded trees of earlier runs while leaving the coding agent itself unchanged ^[5]. Its GPU kernel experiments used four KernelBench tasks, VGG16, LayerNorm, ConvDiv and ConvMax, with Gemini-3.1 Pro as the coding agent, candidates scored by inverse runtime (1/ms) subject to correctness checks against the reference implementation, and a fixed-policy baseline run under the same protocol and initialization ^[5]. The paper reports that on VGG16 and LayerNorm Dream-RSI reached comparable kernel performance with 2.43x and 1.79x fewer generations, and that on ConvDiv and ConvMax it reached 2.09x and 1.44x higher performance under comparable discovery budgets, with trajectories plotted out to roughly 1,000 generations ^[5]. These are the authors' own figures against their own baseline on four tasks, not fast_p scores over the full benchmark, so they are not comparable with the results in the section above.

Significance

KernelBench helped define a distinct line of evaluation focused on AI systems that write performant low-level code rather than ordinary application software. Its combination of an executable correctness check, a real speedup measurement on actual GPUs, and the tunable fast_p metric gave the research community a shared yardstick for the fast-moving "AI writing CUDA" trend that gathered momentum through 2025 ^[1][2][4].

The benchmark became a common target for agentic kernel-generation efforts and test-time methods, in which a model is wrapped in a loop that compiles, runs, profiles, and revises its own kernels. NVIDIA's DeepSeek-R1 workflow is one prominent example, and a series of subsequent academic systems have used KernelBench to report progress on automated kernel optimization ^[4]. Because gains on the benchmark map onto faster real kernels, improvements have direct practical value for the cost and speed of training and serving large models.

KernelBench also has clear limitations that its authors acknowledge. Tasks are evaluated only on fixed input shapes, so a kernel need not generalize to other sizes, and correctness is judged by numerical tolerance on a handful of random inputs rather than by formal verification. The lack of a prescribed train/test split means results depend on the protocol each group chooses, which can complicate comparison. There is also an inherent tension between maximizing correctness and maximizing speed, which the fast_p framing makes visible but does not resolve. Despite these caveats, KernelBench established performance-aware kernel generation as a measurable, reproducible challenge and remains a reference benchmark for the field ^[1][2].

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15Ouyang, Anne; Guo, Simon; Arora, Simran; Zhang, Alex L.; Hu, William; Ré, Christopher; Mirhoseini, Azalia. "KernelBench: Can LLMs Write Efficient GPU Kernels?" arXiv:2502.10517, February 14, 2025. arxiv.org/...2502.10517
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16Scaling Intelligence Lab, Stanford University. "KernelBench: Can LLMs Write Efficient GPU Kernels?" (project page and blog). scalingintelligence.stanford.edu/...kernelbench and scalingintelligence.stanford.edu/...kernelbench
3. ^International Conference on Machine Learning (ICML) 2025 poster: "KernelBench: Can LLMs Write Efficient GPU Kernels?" icml.cc/...43517
4. ^1 ^2 ^3 ^4NVIDIA Developer Blog. "Automating GPU Kernel Generation with DeepSeek-R1 and Inference Time Scaling." February 12, 2025. developer.nvidia.com/...and-inference-time-scaling
5. ^1 ^2 ^3Zheng, Tong; Wu, Xidong; Zhang, Zheng; et al. "Dream-RSI: Recursive Self-Improvement through Evolving Worlds." arXiv:2609.14858, September 14, 2026 (Section 4.3 and Figure 4). arxiv.org/...2609.14858
6. ^ScalingIntelligence/KernelBench, GitHub repository README (levels, fast_p definitions, baseline timing guidance). github.com/...KernelBench