LiveCodeBench
LiveCodeBench is a holistic and contamination-free benchmark for evaluating large language models on code, first released in March 2024 by researchers at UC Berkeley, MIT, and Cornell led by Naman Jain.[1] Unlike static benchmarks, it continuously collects fresh problems from competitive programming platforms (LeetCode, AtCoder, and Codeforces) and tags each problem with its publication date, so a model can be evaluated only on problems released after its training cutoff to guarantee an uncontaminated score.[1][2] The paper describes the benchmark as "a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms."[1] It was published as a conference paper at ICLR 2025 and has become one of the most widely cited measures of coding ability in language models, with developer-reported scores reaching 93.5% Pass@1 (DeepSeek V4 Pro Max, as listed by the llm-stats aggregator) by 2026.[10][12][16]
| LiveCodeBench | |
|---|---|
| Overview | |
| Full name | Live Code Benchmark |
| Abbreviation | LCB |
| Description | A holistic and contamination-free evaluation benchmark for code LLMs with continuous updates |
| Release date | 2024-03 |
| Latest version | v6 |
| Benchmark updated | 2025-05 |
| Authors | Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, Ion Stoica |
| Organization | UC Berkeley, MIT, Cornell |
| Technical Details | |
| Type | Code Generation, Code Understanding, Multi-task |
| Modality | Text (Code) |
| Task format | Code generation, self-repair, test output prediction, code execution |
| Number of tasks | 1,055+ (as of v6) |
| Total examples | 1,055+ |
| Evaluation metric | Pass@1, Pass@5, Execution accuracy |
| Domains | Competitive programming |
| Languages | Python, C++, Java, and others |
| Performance | |
| Human performance | Variable by task and difficulty |
| Baseline | ~20-30% (smaller models) |
| SOTA score | 93.5% (Pass@1, self-reported) |
| SOTA model | DeepSeek V4 Pro Max |
| SOTA date | 2026-04 |
| Saturated | No |
| Resources | |
| Website | Official website |
| Paper | Paper |
| GitHub | Repository |
| Dataset | Hugging Face |
| License | MIT |
| Venue | ICLR 2025 |
What is LiveCodeBench?
LiveCodeBench is a holistic and contamination-free artificial intelligence benchmark for evaluating large language models on code-related tasks.[1] First released in March 2024 by researchers at UC Berkeley, MIT, and Cornell, the benchmark tackles one of the most persistent problems in code model evaluation: data contamination.[1][5] Instead of relying on a fixed set of problems that models may have encountered during training, LiveCodeBench continuously collects fresh problems from competitive programming platforms, including LeetCode, AtCoder, and Codeforces.[1][2] Each problem is tagged with its publication date, allowing evaluators to test models only on problems released after a given model's training cutoff.[1] The initial release (v1) contained 400 problems published between May 2023 and March 2024.[3] The benchmark was published as a conference paper at ICLR 2025[10] and has since become one of the most widely cited benchmarks for measuring coding ability in language models.
Why was LiveCodeBench created?
By early 2024, the field of code-generating language models had grown rapidly. Benchmarks such as HumanEval, MBPP, and APPS served as standard evaluation tools, but they suffered from a fundamental limitation: they were static. Once a benchmark's problems became publicly available, there was no way to prevent those problems from appearing in future training datasets. As a result, models trained on data collected after a benchmark's release could achieve inflated scores without genuinely improving at coding tasks. This phenomenon, known as data contamination or benchmark leakage, undermined the reliability of published results and made it difficult to compare models fairly.[1] The authors note that "as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities."[1]
Several observations highlighted the severity of the contamination problem. Fine-tuned open-source models sometimes achieved high marks on HumanEval while performing poorly on genuinely unseen problems, suggesting that their strong benchmark scores reflected memorization rather than generalization.[1][5] Meanwhile, closed-source models like GPT-4 maintained more consistent performance across old and new problems, indicating that their capabilities were more robust.
Beyond contamination, existing benchmarks also had a narrow evaluation scope. Most focused exclusively on code generation: given a natural language description, produce a working program. Real-world software development, however, involves debugging, understanding existing code, predicting program behavior, and iterating on failed attempts. A benchmark that measured only generation captured just one facet of what it means to be a capable coding model.[1]
These two problems, contamination and narrow scope, motivated the creation of LiveCodeBench. The researchers set out to build a benchmark that would remain fresh indefinitely through continuous updates and that would measure a broader set of coding capabilities through multiple evaluation scenarios.
Who created LiveCodeBench?
LiveCodeBench was developed by a team spanning three universities:[1][5]
| Author | Affiliation | Role |
|---|---|---|
| Naman Jain | UC Berkeley | Co-lead author |
| King Han | UC Berkeley | Second author |
| Alex Gu | MIT | Contributing author |
| Wen-Ding Li | Cornell | Contributing author |
| Fanjia Yan | UC Berkeley | Contributing author |
| Tianjun Zhang | UC Berkeley | Contributing author |
| Sida Wang | No affiliation listed in the paper | Contributing author |
| Armando Solar-Lezama | MIT | Senior advisor |
| Koushik Sen | UC Berkeley | Senior advisor |
| Ion Stoica | UC Berkeley | Senior advisor |
The project emerged from the UC Berkeley Sky Computing Lab, a research group led by Ion Stoica that focuses on large-scale computing systems and AI infrastructure. The involvement of Armando Solar-Lezama, a leading figure in program synthesis research at MIT, brought additional expertise in automated code reasoning.
How are problems collected?
Source platforms
LiveCodeBench draws problems from three major competitive programming platforms, each contributing a different style of problem:[1]
| Platform | Problem style | Contest frequency | Difficulty rating system | Test case availability |
|---|---|---|---|---|
| LeetCode | Algorithm and data structure puzzles | Weekly and biweekly contests | Easy, Medium, Hard | Hidden tests not public; tests built with generated inputs |
| AtCoder | AtCoder Beginner Contest (abc) problems only | Weekly abc contests | Numeric rating (abc problems rated above 500 excluded) | Full test suites provided |
| Codeforces | Division 3 and Division 4 rounds only | Regular rounds | Numeric difficulty rating | Partial (long tests are truncated) |
These platforms host regular programming contests where thousands of participants solve problems under time pressure. The large number of participants ensures that each problem is thoroughly vetted for clarity, correctness, and appropriate difficulty before it enters the benchmark.
Scraping and filtering pipeline
The LiveCodeBench team built custom HTML scrapers for each of the three platforms. These scrapers collect:[1]
- The natural language problem statement
- Example input-output pairs
- Hidden test cases (where available)
- Contest metadata, including the exact date and time of the contest
- Difficulty ratings assigned by the platform
After scraping, the problems go through a filtering step. Problems that require image interpretation are excluded, since the benchmark focuses on text-based code evaluation. Problems that accept multiple valid outputs are also excluded to simplify automated evaluation. For LeetCode, whose hidden tests are not public, and for Codeforces, where long test inputs are truncated, the researchers build input generators from the problem specifications using GPT-4-Turbo (semi-automatically for the nine Codeforces problems) and pair the generated inputs with collected ground-truth solutions.[1]
Difficulty classification
Each platform has its own difficulty rating system. To create a unified difficulty scale, the researchers map platform-specific ratings into three tiers:[1]
| Tier | LeetCode | AtCoder (abc only) | Codeforces (Div 3 and 4 only) |
|---|---|---|---|
| Easy | Easy label | Rating 0-199 | Rating 800 |
| Medium | Medium label | Rating 200-399 | Rating 801-1000 |
| Hard | Hard label | Rating 400-500 | Rating 1001-1300 |
Problems rated above a certain difficulty threshold are excluded from the benchmark because they are too difficult for even the strongest models, which would introduce noise into the evaluation without providing useful signal.
Test case statistics
On average, each problem in the benchmark comes with approximately 17 test cases. The number varies by platform:[1]
| Platform | Average test cases per problem |
|---|---|
| LeetCode | 19.0 |
| AtCoder | 15.6 |
| Codeforces | 11.1 |
How has the dataset grown over time?
LiveCodeBench is designed to grow over time as new contest problems become available. The team periodically releases updated versions of the dataset:[3][14]
| Version | Release date | Problem count | Coverage period |
|---|---|---|---|
| v1 | March 2024 | 400 | May 2023 to March 2024 |
| v2 | May 2024 | 511 | May 2023 to May 2024 |
| v3 | July 2024 | 612 | May 2023 to July 2024 |
| v4 | September 2024 | 713 | May 2023 to September 2024 |
| v5 | January 2025 | 880 | May 2023 to January 2025 |
| v6 | May 2025 | 1,055 | May 2023 to April 2025 |
The initial v1 release contained 400 problems.[1] In the accompanying paper, the researchers reported results on a dataset of 511 problems (later designated v2), with a difficulty distribution of 182 Easy, 206 Medium, and 123 Hard problems.[1] Each subsequent release adds problems from contests that occurred after the previous version's cutoff date.[3]
The v5 and v6 evaluation windows
By 2025, the community converged on reporting Pass@1 on problems released between August 2024 and the version cutoff.[4][14] For v5 this typically meant August 2024 to January 2025; v6 standardized on August 2024 to May 2025 (approximately 454 fresh problems for contamination-free testing of any model with a training cutoff of July 2024 or earlier).[14] The official leaderboard lets users choose a time window with a slider and highlights in red models that are likely contaminated on some fraction of the problems in the selected window, judged from model release dates and performance variation.[4]
What tasks does LiveCodeBench evaluate?
LiveCodeBench goes beyond simple code generation by evaluating models on four distinct tasks.[1][5] These tasks were chosen because they represent useful components in real-world code LLM workflows and each has a clear, automated evaluation metric.
Code generation
This is the primary task and the one most directly comparable to benchmarks like HumanEval. The model receives a natural language problem description, including example input-output pairs, and must produce a complete program that passes all hidden test cases.[1]
For instruction-tuned models, the evaluation uses zero-shot prompts. For base models (those without instruction fine-tuning), a one-shot example is provided to demonstrate the expected output format.
The primary metric is Pass@1, defined as the fraction of problems for which the model generates a correct solution on its first attempt. The researchers also compute Pass@5, which measures whether at least one of five generated solutions is correct. To estimate these metrics, 10 candidate solutions are sampled per problem at temperature 0.2 with top_p set to 0.95.[1]
Self-repair
The self-repair task evaluates a model's ability to debug and fix broken code.[1] The evaluation proceeds in two stages:
- The model first attempts to generate a solution (as in the code generation task).
- If the generated code fails any test case, the model receives the original problem description, its incorrect solution, the failing test case, and the error message or execution feedback.
- The model must then produce a corrected version of the program.
The evaluation metric is the combined Pass@1 after the repair step: a problem counts as solved if the model either generated a correct solution on the first attempt or successfully repaired its initial attempt. This mirrors the real-world debugging workflow where a developer writes code, sees test failures, and iterates on the solution.
Code execution
The code execution task tests whether a model can mentally trace through code and predict its output.[1] The model receives a program snippet along with a specific input and must predict the exact output.
The dataset for this task uses approximately 2,000 correct human-submitted LeetCode solutions. These solutions go through compile-time and runtime filters to ensure they have reasonable complexity and produce deterministic outputs. The final dataset consists of 479 samples drawn from 85 problems.[1]
Two prompting strategies are used:
- A 2-shot prompt without chain-of-thought reasoning
- A 1-shot prompt with detailed chain-of-thought steps showing how to trace through code manually
The paper found that closed-source models benefited from chain-of-thought prompting on this task, while open-source models sometimes performed worse with chain-of-thought, possibly due to difficulties maintaining coherent multi-step reasoning.[1]
Test output prediction
In the test output prediction task, the model receives a problem description and a test input, but not a solution.[1] It must predict the correct output purely from its understanding of the problem.
This task is simpler than code generation in one sense (no code needs to be written) but requires deep comprehension of the problem logic. The dataset contains 442 instances drawn from 181 LeetCode problems. Evaluation uses a zero-shot prompt that asks the model to complete an assertion with the expected output value.
Task comparison table
| Task | Input given to model | Expected output | Evaluation metric | Dataset size | Real-world analogy |
|---|---|---|---|---|---|
| Code Generation | Problem description + examples | Working program | Pass@1, Pass@5 | 511 problems (v2) | Writing new code from a specification |
| Self-Repair | Problem + buggy code + error feedback | Corrected program | Combined Pass@1 | Same as code generation | Debugging a failing test |
| Code Execution | Program + input | Predicted output | Exact match accuracy | 479 samples from 85 problems | Code review and tracing |
| Test Output Prediction | Problem description + test input | Expected output | Exact match accuracy | 442 instances from 181 problems | Understanding requirements without coding |
How does LiveCodeBench prevent contamination?
Temporal windowing
The core contamination prevention mechanism in LiveCodeBench is temporal windowing.[1] Every problem in the benchmark is tagged with the date it was published on its source platform. When evaluating a model, the evaluator can filter problems to include only those released after the model's known training data cutoff. This means that even if older problems have leaked into training data, the evaluation can still produce an uncontaminated score by restricting the evaluation window to newer problems. The paper reports a stark performance drop for DeepSeek, GPT-4o and Codestral on LeetCode problems released after their release or cutoff dates and concludes that "time-segmented evaluations prove effective for performing fair comparisons."[1]
For example, if a model's training data ends in September 2023, the evaluator can restrict the evaluation to problems released from October 2023 onward. This guarantees that the model has never seen any of the evaluation problems during training.
Contamination detection
Beyond prevention, LiveCodeBench also serves as a contamination detection tool. By comparing a model's performance on problems from different time periods, researchers can identify suspicious performance patterns that suggest data leakage.[1]
The original paper identified several notable contamination patterns:[1]
| Model | Observed pattern | Likely explanation |
|---|---|---|
| DeepSeek-Coder | Sharp performance drop on LeetCode problems released after August 2023 | Training data likely included LeetCode problems up to the model's release date |
| GPT-4o | Performance decline on problems released after November 2023 | Training cutoff alignment |
| Codestral | Pass@1 dropped from 36.5% on older problems to 28.3% on newer problems | Training data contamination on earlier problems |
| Claude 3 | No drastic month-to-month variation around its August 2023 cutoff | No clear contamination signal (same finding for GPT-4-Turbo, Gemini-Pro and Mistral-Large) |
These findings demonstrated that contamination is not limited to open-source models; even major closed-source models show signs of having been exposed to competitive programming problems that appeared online before their training cutoff dates.
Contamination across platforms
Interestingly, the contamination effect was strongest for LeetCode problems and weaker for AtCoder and Codeforces problems.[1] This likely reflects the fact that LeetCode problems and solutions are more widely shared on the open web (through blog posts, GitHub repositories, and discussion forums), making them more likely to appear in web-scraped training data.
Evaluation setup and implementation
Generation parameters
All evaluations use a consistent set of generation parameters:[1]
| Parameter | Value |
|---|---|
| Number of samples per problem (n) | 10 |
| Temperature | 0.2 |
| Top-p | 0.95 |
| Metric computation | Unbiased estimator for Pass@1 and Pass@5 |
The low temperature (0.2) ensures relatively deterministic outputs while still allowing some diversity across samples. The unbiased estimator for Pass@k, originally introduced by the Codex paper from OpenAI, computes the metric from the 10 samples without requiring expensive repeated sampling.
Execution environment
Generated code is executed in a sandboxed environment to validate functional correctness. The researchers use a modified version of the checker from the APPS benchmark, with identified edge cases fixed and the checker simplified for the LiveCodeBench dataset.[1]
Installation and usage
LiveCodeBench is open-source under the MIT license.[3] Usage, per the repository README:
# Clone the repository and install (the README recommends uv)
git clone https://github.com/LiveCodeBench/LiveCodeBench.git
cd LiveCodeBench
uv venv --python 3.11
source .venv/bin/activate
uv pip install -e .
# Generate and evaluate code for a given dataset release
python -m lcb_runner.runner.main --model {model_name} --scenario codegeneration --evaluate --release_version release_v6
Scores for a specific time window are computed from the saved evaluation file:
python -m lcb_runner.evaluation.compute_scores --eval_all_file {saved_eval_all_file} --start_date 2024-08-01
Key findings from the original paper
The original LiveCodeBench paper evaluated 18 base LLMs and 34 instruction-tuned LLMs, making it one of the largest evaluation studies of code models on competitive programming problems at the time of publication.[1][5]
Model rankings (original paper, 2024)
The project's original write-up named GPT-4-Turbo and Claude-3-Opus as the best performers across scenarios and DS-Ins-33B and Phind-34B as the strongest open models.[2][5] In the June 2024 revision of the paper (511 problems, May 2023 to May 2024), GPT-4o-2024-05-13 had the highest overall code generation Pass@1 (49.1%), ahead of GPT-4-Turbo-2024-04-09 (45.6%), Gemini-Pro-1.5-May (40.7%) and Claude-3-Opus (37.8%), and Llama 3 Instruct 70B (30.1%) led the open models.[1] The models highlighted in the original analysis were:
| Model | Pass@1 (code generation) | Category |
|---|---|---|
| GPT-4-Turbo | Highest among all models | Closed-source |
| Claude 3 Opus | Second highest, close to GPT-4-Turbo | Closed-source |
| GPT-4 | Strong but slightly below Turbo variant | Closed-source |
| DeepSeek-Coder-Instruct 33B | Best open-source model | Open-source |
| Llama 3 Instruct 70B | Strong open-source performer | Open-source |
The paper found a significant performance gap between closed-source and open-source models. Only the strongest instruction-tuned variants of models with more than 30 billion parameters (such as Llama 3 Instruct 70B, Mixtral, and DeepSeek-Instruct 33B) came close to bridging this gap.[1]
HumanEval vs. LiveCodeBench correlation
A key finding was the moderate correlation (r = 0.72) between HumanEval scores and LiveCodeBench scores.[1] The researchers identified two distinct clusters of models:
- Cluster 1: Base models and closed-source models that performed similarly on both benchmarks, indicating genuine coding ability.
- Cluster 2: Fine-tuned open-source models that scored well on HumanEval but poorly on LiveCodeBench, suggesting overfitting to HumanEval's specific problem set.
For instance, DeepSeek-Instruct 33B trailed GPT-4-Turbo by only 4.3 percentage points on HumanEval+ but by 16.2 points (69%) on the LiveCodeBench code generation scenario.[1] This disparity highlights how static benchmarks can paint a misleading picture of model capabilities.
Task-specific observations
- The gap between open and closed models widened on self-repair and test output prediction compared with code generation.
- GPT-4-Turbo extended its lead over GPT-4 in the self-repair scenario.
- Code execution with chain-of-thought prompting helped closed-source models but hurt some open-source models.
- Test output prediction performance varied widely, with some models showing unexpectedly strong problem comprehension despite weaker code generation scores.[1]
What is the current LiveCodeBench leaderboard (2026)?
As LiveCodeBench has continued to be updated, it has become a standard evaluation benchmark tracked by multiple leaderboard platforms. The official leaderboard's problem set ends in April 2025 and does not include the models below. The following table compiles figures from third-party aggregators as of mid-2026; the scores come from different sources and harnesses (developer self-reports collected by llm-stats, and independent runs by Artificial Analysis) and are not strictly comparable:[9][12][14]
| Rank | Model | Organization | Pass@1 score |
|---|---|---|---|
| 1 | DeepSeek V4 Pro Max | DeepSeek | 93.5% |
| 2 | Gemini 3 Pro Preview | Google DeepMind | 91.7% |
| 3 | DeepSeek V4 Flash Max | DeepSeek | 91.6% |
| 4 | Gemini 3 Flash Preview (Reasoning) | Google DeepMind | 90.8% |
| 5 | DeepSeek V3.2 Speciale | DeepSeek | 89.6% |
| 6 | Claude Opus 4.6 (Max) | Anthropic | 88.8% (reported by DeepSeek for comparison) |
| 7 | Step-3.5-Flash | StepFun | 86.4% |
| 8 | DeepSeek V3.2 (Thinking) | DeepSeek | 83.3% |
| 9 | MiniMax M2 | MiniMax | 83.0% |
| 10 | LongCat-Flash-Thinking-2601 | Meituan | 82.8% |
| 11 | Nemotron 3 Super (120B A12B) | NVIDIA | 81.2% |
| 12 | Grok 3 Mini | xAI | 80.4% |
| 13 | Grok 4 Fast | xAI | 80.0% |
| 14 | Grok 3 | xAI | 79.4% |
| 15 | Grok 4 Heavy | xAI | 79.4% |
DeepSeek V4 Pro Max, released by DeepSeek on April 24, 2026, holds the highest score on the llm-stats list (93.5% Pass@1, as reported in DeepSeek's own model card).[12][16] The V4 family is a Mixture-of-Experts architecture released under the MIT license, extending a streak that began with V3.2 Thinking. Google DeepMind's Gemini 3 series followed closely.[9] GPT-5.5, released by OpenAI in April 2026, did not have a LiveCodeBench figure on the aggregators cited here as of mid-May 2026. Qwen3-Max from Alibaba is similarly absent from the official site, though third-party aggregators list partial results.
Lab-specific 2026 highlights
| Lab | Flagship model | LiveCodeBench Pass@1 (v6 window) | Notes |
|---|---|---|---|
| DeepSeek | DeepSeek V4 Pro Max | 93.5% (self-reported) | Highest score on the llm-stats list |
| Google DeepMind | Gemini 3 Pro Preview | 91.7% | Strongest closed-source model on Artificial Analysis |
| Anthropic | Claude Opus 4.6 (Max) | 88.8% (reported by DeepSeek) | Anthropic has not published an official LCB figure |
| OpenAI | GPT-5.5 | Not officially reported | Launch focused on internal coding-agent metrics rather than LCB |
| xAI | Grok 4 Heavy | 79.4% | Grok 4 is listed at 79.0% |
| Alibaba | Qwen3-Max | Not on official leaderboard | Third-party evaluations indicate competitive performance |
The mid-2026 picture is one of intense competition between Chinese frontier labs (DeepSeek, MiniMax, Meituan, Qwen) and the leading US labs (Google DeepMind, Anthropic, OpenAI, xAI).[9][12] The official site continues to insist on the contamination-controlled window for ranking, while third-party aggregators offer broader pictures.
Why do leaderboards disagree?
LiveCodeBench scores reported in 2025 and 2026 vary noticeably depending on the source:[9][11][12]
| Source | Methodology highlights | Comparability |
|---|---|---|
| Official LiveCodeBench | Hidden and generated tests, Pass@1 over 10 samples at temperature 0.2; problems up to April 2025 | Baseline |
| Artificial Analysis | Independent evaluations run by Artificial Analysis | Different harness from the official one |
| llm-stats.com | Collects scores self-reported by model developers | Unverified; windows and settings vary |
| PricePerToken | Republishes Artificial Analysis data alongside API prices | Same as Artificial Analysis |
| BenchLM | Composite coding score combining LiveCodeBench with other benchmarks | Not a direct LiveCodeBench score |
These differences can change rankings. Gemini 3 Pro Preview (High) holds the top spot in Artificial Analysis's independent runs with 91.7%, while DeepSeek V4 Pro Max leads the self-reported llm-stats list at 93.5%; neither model appears on the official leaderboard, whose problem set ends in April 2025.[4][9][12]
The rise of reasoning models
A significant trend visible in the 2025-2026 leaderboard is the dominance of reasoning-enabled models. Models with explicit chain-of-thought or "thinking" modes (such as DeepSeek V3.2 Thinking, DeepSeek V4 Pro Max, Gemini 3 Pro reasoning variants, and Claude Opus 4.7 with extended thinking) consistently outperform their non-reasoning counterparts.[9][12] The improvement is most pronounced on problems that require structured algorithmic thinking, such as combinatorics and dynamic programming. However, reasoning provides limited gains on problems that require careful observation of edge cases or complex pattern matching.
What is LiveCodeBench Pro?
LiveCodeBench Pro is a separate, harder benchmark of 584 elite competitive programming problems drawn exclusively from Codeforces, ICPC, and IOI contests, with every problem annotated by Olympiad medalists.[6] It was introduced in June 2025 in a paper titled "LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?" (arXiv: 2506.11928).[6] While the original LiveCodeBench targets a broad range of difficulty levels, LiveCodeBench Pro focuses specifically on challenging problems from elite competitive programming contests, and a team of Olympiad medalists conducts a line-by-line analysis of every failed model submission.[6]
Authors and institutions
LiveCodeBench Pro was developed by a larger cross-institutional team than the original LiveCodeBench. The authors include Zihan Zheng and Saining Xie of New York University; Zerui Cheng, Stanley Wei, Zixuan Wang, Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, and Pramod Viswanath of Princeton University; Zeyu Shen and Shang Zhou of UC San Diego; Kaiyuan Liu of the University of Washington; Hansen He of Canyon Crest Academy; Dongruixuan Li of the University of Waterloo; Hangyi Hao of McGill University; Jianzhu Yao of Princeton University; Peiyao Sheng of the Sentient Foundation; and Jingbo Shang of UC San Diego.[6] The annotation team draws heavily on former International Olympiad in Informatics (IOI), ICPC World Finals, and national-team medalists, which is the basis for the paper's claim of medalist-level review.
How does LiveCodeBench Pro differ from the original?
| Feature | LiveCodeBench | LiveCodeBench Pro |
|---|---|---|
| Problem sources | LeetCode, AtCoder, Codeforces | Codeforces, ICPC, IOI |
| Problem count | 1,055+ (v6) | 584 |
| Difficulty focus | Easy to Hard | Medium to Expert (Olympiad-level) |
| Rating system | Platform-specific difficulty tiers | Codeforces-style Elo ratings |
| Annotation | Automated with manual review | Olympiad medalist annotations |
| Error analysis | Automated | Line-by-line medalist review of failures |
Elo rating system
LiveCodeBench Pro uses Bayesian Elo ratings to assess model performance, making scores directly comparable to human competitive programmers on platforms like Codeforces.[6][8] Each LLM is treated as a virtual Codeforces contestant, with ratings fit via Bayesian maximum-a-posteriori (MAP) Elo on the standard Codeforces scale, which runs from roughly 800 (novice) to roughly 3,800 (top human).[8] Problems span three tiers based on Elo:
| Tier | Elo range | Description |
|---|---|---|
| Easy | Up to 2000 | Standard competitive programming problems |
| Medium | 2000 to 3000 | Advanced algorithmic challenges |
| Hard | Above 3000 | Olympiad-level problems requiring deep mathematical reasoning |
Findings from the original release
LiveCodeBench Pro revealed stark limitations in current models. Without external tools, the best model achieved only 53% Pass@1 on medium-difficulty problems and 0% on hard problems.[6] Allowing multiple attempts (Pass@10) substantially improved performance, with some models gaining over 500 Elo points. The benchmark also introduced a cognitive-focus taxonomy, labeling problems as "knowledge-heavy", "logic-heavy" or "observation-heavy" (problems hinging on a single key insight). Models did well on knowledge-heavy and logic-heavy problems but struggled on observation-heavy ones.[6]
2026 Elo leaderboard
LiveCodeBench Pro has emerged as the more discriminating benchmark for frontier models, since the original LiveCodeBench begins to compress at the top. As of November 2026, the llm-stats aggregator listed these developer-reported Elo ratings:[15]
| Rank | Model | Lab | Pro Elo rating |
|---|---|---|---|
| 1 | Gemini 3.1 Pro | Google DeepMind | 2,887 |
| 2 | Gemini 3 Pro | Google DeepMind | 2,439 |
| 3 | Gemini 3 Flash | Google DeepMind | 2,316 |
| 4 | Seed 1.8 | ByteDance | 1,930 |
A 2,887 Elo rating would place Gemini 3.1 Pro in the International Grandmaster band (2,600-2,999) of the Codeforces rating scale, below the Legendary Grandmaster threshold of 3,000.[15][19] Even at this level, models still fail completely on the hardest Olympiad problems that require novel algorithmic insight.
GSO: Global Software Optimization
In May 2025, researchers including LiveCodeBench lead author Naman Jain launched GSO (Global Software Optimization), a separate benchmark focused on software optimization rather than algorithmic problem-solving.[17] GSO presents models with codebases and performance tests, tasking them to improve runtime efficiency. The benchmark includes 102 optimization tasks across 10 codebases and uses the Opt@1 metric, measuring the fraction of tasks where a single attempt achieves at least 95% of the speedup that a human expert achieved. In November 2025, GSO added a "Hack Detector" that penalizes deceptive optimizations such as memoization tricks or test harness hijacking.[18]
How does LiveCodeBench compare with other code benchmarks?
LiveCodeBench occupies a specific niche in the landscape of code evaluation benchmarks:
| Benchmark | Problem type | Size | Dynamic updates | Contamination prevention | Tasks evaluated |
|---|---|---|---|---|---|
| HumanEval | Hand-written Python puzzles | 164 | No | None | Code generation only |
| MBPP | Crowd-sourced Python problems | ~1,000 | No | None | Code generation only |
| APPS | Algorithmic problems | 10,000 | No | None | Code generation only |
| CodeContests | Competition problems | ~13,000 | No | None | Code generation only |
| SWE-bench | Real GitHub issues | 2,294 | No | None | End-to-end software engineering |
| BigCodeBench | Complex function-level tasks | 1,140 | No | None | Code generation, function completion |
| LiveCodeBench | Competition problems | 1,055+ | Continuous | Temporal windowing | Generation, repair, execution, prediction |
LiveCodeBench's primary advantages are its continuous update mechanism and multi-task evaluation scope. Its primary limitation compared to benchmarks like SWE-bench is that it focuses on self-contained algorithmic problems rather than real-world software engineering tasks that require navigating large codebases, understanding project context, and writing tests.
Position in the modern benchmark stack
By 2026, frontier-model evaluations typically report a battery of coding metrics rather than a single number. A common configuration includes function-level benchmarks (HumanEval+, MBPP+), competitive programming (LiveCodeBench v6), elite programming (LiveCodeBench Pro), real codebase work (SWE-bench Verified), optimization (GSO), and agent workflows (Aider Polyglot, SWE-Agent). In this stack, LiveCodeBench and LiveCodeBench Pro together occupy the algorithmic-reasoning slot. They sit between the function-level benchmarks that have largely saturated (HumanEval is above 95% for most frontier models) and the agent-style benchmarks where evaluation cost and noise are much higher.[13]
Adoption and impact
Since its release, LiveCodeBench has seen broad adoption across both academia and industry:
- Labs including DeepSeek, Alibaba's Qwen team, xAI and NVIDIA have reported LiveCodeBench scores for their models, and Google DeepMind reports LiveCodeBench Pro.[12][15]
- The Hugging Face platform hosts an official LiveCodeBench leaderboard that allows researchers to submit and compare results.[5][7]
- The benchmark's temporal windowing approach has influenced the design of other contamination-resistant benchmarks, including LiveBench (a general-purpose LLM benchmark) and LiveCodeBench Pro.[6]
- The paper was accepted at ICLR 2025, one of the top venues in machine learning research.[10]
- Independent aggregators such as Artificial Analysis, llm-stats.com, and PricePerToken treat LiveCodeBench as a primary signal for coding-quality rankings.[9][11][12]
How has the methodology evolved since release?
LiveCodeBench's methodology has not stayed static. The maintainers have iterated on the harness, problem-curation pipeline, and reporting conventions in response to community feedback and the emergence of new model capabilities.
Errata and problem quality
The repository keeps an ERRATA file documenting known problem issues, such as problems that accept multiple valid outputs but are graded against one, interactive problems the harness cannot run, and problems with erroneous test cases; the maintainers say they use this feedback to improve problem selection as the benchmark is updated.[3]
Criticism and ongoing debates
Like any widely-adopted benchmark, LiveCodeBench has attracted scrutiny. Several debates have shaped how the community interprets results:
| Debate | Position 1 | Position 2 |
|---|---|---|
| Competitive programming vs. real coding | Olympiad-style problems are a poor proxy for software engineering | Algorithmic reasoning still correlates with coding ability and is harder to game |
| Contamination through fine-tuning | Open-weights models fine-tuned on Codeforces editorial archives game the benchmark | The temporal window ensures fairness regardless of fine-tuning data |
| Closed vs. open prompts | Per-model prompt engineering inflates closed-source numbers | Standardized prompts disadvantage models with different chat templates |
| Reasoning models on Pro | The Elo system over-rewards long deliberation impractical in production | Inference-time compute is a real capability that deserves measurement |
A frequent 2025-2026 criticism is that strong LiveCodeBench scores have become necessary but not sufficient.[13] Models that lead on LiveCodeBench but lag on SWE-bench Verified or agent harnesses are viewed as having a competitive-programming bias rather than well-rounded engineering skill.
Limitations
Despite its contributions, LiveCodeBench has several recognized limitations:[1]
| Limitation | Description | Impact |
|---|---|---|
| Platform dependency | Relies on external contest platforms for new problems | Data availability is tied to contest schedules |
| Algorithmic focus | Problems are primarily algorithmic puzzles from competitions | May not reflect practical software engineering tasks |
| Language bias | While multiple languages are supported, most evaluation focuses on Python | Limited assessment of language-specific capabilities |
| Execution cost | Sandboxed execution of generated code requires significant compute | Resource-intensive evaluation |
| Difficulty ceiling | Problems above a certain difficulty are excluded | Does not test the absolute upper bound of model capability |
| Update irregularity | New problems depend on contest schedules across three platforms | Uneven temporal coverage |
| Reasoning compute confound | Reasoning-mode scores depend on token budgets and sampling temperature | Pass@1 numbers across labs may not be directly comparable |
Future directions
The original paper describes LiveCodeBench as an extensible framework and names input generation, program summarization and optimization as settings that could be added as new scenarios.[1] The team's later GSO benchmark took up software optimization as a separate project.[17]
Continue reading
- HumanEvalHumanEval is a benchmark for measuring whether a code-generating large language model can complete short Python functions so that they pass unit tests. It contains 164 hand-written tasks.Mentioned 12 times above
- MBPPMBPP (Mostly Basic Python Problems) is a code generation benchmark of 974 crowd-sourced Python programming tasks designed to be solvable by entry-level programmers, introduced by Jacob Austin, Augustus Odena…Mentioned 4 times above
- SWE-benchSWE-bench is an execution-based benchmark for evaluating whether a language-model system can resolve real software issues.Mentioned 4 times above
- Data contaminationData contamination is the presence of benchmark evaluation data in the corpus used to train a modelMentioned 3 times above
- Large Language ModelA large language model (LLM) is an artificial intelligence system built on a transformer neural network with billions to trillions of parameters, trained on massive text corpora to predict the next token in a…Mentioned twice above
- Program SynthesisProgram synthesis is the task of automatically constructing a program that satisfies a specification expressed at a higher level than the code itself: a logical formula, a set of input-output examples, a…Listed in See also
More from See also
More to read
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., & Stoica, I. (2024). "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code." *arXiv preprint arXiv:2403.07974*. Published as a conference paper at ICLR 2025. arxiv.org/...2403.07974
- ^1 ^2 ^3LiveCodeBench Official Website. livecodebench.github.io
- ^1 ^2 ^3 ^4 ^5LiveCodeBench GitHub Repository. github.com/...LiveCodeBench
- ^1 ^2 ^3LiveCodeBench Leaderboard. livecodebench.github.io/leaderboard
- ^1 ^2 ^3 ^4 ^5 ^6 ^7Hugging Face Blog. "Introducing the LiveCodeBench Leaderboard: Holistic and Contamination-Free Evaluation of Code LLMs." huggingface.co/...leaderboard-livecodebench
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Zheng, Z., Cheng, Z., Shen, Z., Zhou, S., Liu, K., He, H., Li, D., Wei, S., Hao, H., Yao, J., Sheng, P., Wang, Z., Chai, W., Korolova, A., Henderson, P., Arora, S., Viswanath, P., Shang, J., & Xie, S. (2025). "LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?" *arXiv preprint arXiv:2506.11928*. arxiv.org/...2506.11928
- ^LiveCodeBench on Hugging Face. huggingface.co/livecodebench
- ^1 ^2LiveCodeBench Pro Leaderboard. web.archive.org/...livecodebenchpro.com
- ^1 ^2 ^3 ^4 ^5 ^6 ^7Artificial Analysis LiveCodeBench Leaderboard. artificialanalysis.ai/...livecodebench
- ^1 ^2 ^3LiveCodeBench, ICLR 2025 Proceedings (OpenReview, forum id chfJJYC3iL). openreview.net/forum
- ^1 ^2PricePerToken LiveCodeBench Leaderboard. pricepertoken.com/...livecodebench
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9llm-stats.com LiveCodeBench Benchmark Leaderboard. llm-stats.com/...livecodebench
- ^1 ^2BenchLM.ai Coding Benchmark Tracker. benchlm.ai/coding
- ^1 ^2 ^3 ^4LiveCodeBench v6 Benchmark Leaderboard. llm-stats.com/...livecodebench-v6
- ^1 ^2 ^3LiveCodeBench Pro Benchmark Leaderboard. llm-stats.com/...livecodebench-pro
- ^1 ^2DeepSeek. "DeepSeek-V4-Pro" model card. Hugging Face. huggingface.co/...DeepSeek-V4-Pro
- ^1 ^2Shetty, M., Jain, N., Liu, J., Kethanaboyina, V., Sen, K., & Stoica, I. (2025). "GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents." *arXiv preprint arXiv:2505.23671*. arxiv.org/...2505.23671
- ^GSO Leaderboard and changelog. gso-bench.github.io
- ^"Codeforces." Wikipedia. en.wikipedia.org/...Codeforces
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
9 revisions · v9 · 6,201 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent fact-check of the full article against its cited sources (150 claims, 24 sources fetched); 26 material + 18 minor fixed
Cite this page: AI Wiki. "LiveCodeBench." aiwiki.ai, updated 21 Jun 2026, fact-checked 2 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/livecodebench