BigCodeBench
BigCodeBench is a Python code generation benchmark of 1,140 function-level programming tasks that require composing 723 distinct function calls from 139 libraries across seven domains
AI Benchmarks
Explore AI Benchmarks through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: AI Code Generation
Articles that also belong to these categories. Counts cover all of AI Benchmarks.
All articles (239)Model Evaluation (56)Natural Language Processing (42)Large Language Models (33)AI Agents (28)AI Code Generation (25)AI Safety (25)Machine Learning (22)Computer Vision (19)Multimodal AI (15)Reasoning Models (13)Data & Datasets (11)Artificial Intelligence (8)OpenAI (6)AI Research (5)Healthcare AI (4)Mathematics (4)Open Source AI (4)AI Companies (3)AI Ethics (3)AI Hardware (3)Developer Tools (3)Google DeepMind (3)Image Generation (3)Information Retrieval (3)NVIDIA (3)Robotics (3)Reinforcement Learning (2)Software Development (2)Speech & Audio AI (2)Statistics (2)World Models (2)2025 in artificial intelligence (1)AI Alignment (1)AI Inference (1)AI Models (1)AI in Cybersecurity (1)Autonomous Vehicles (1)Data Science (1)Deep Learning (1)Education AI (1)Embodied AI (1)Enterprise AI (1)Finance AI (1)Generative AI (1)Legal AI (1)Meta AI (1)Microsoft (1)People (1)Physical AI (1)Research Organizations (1)Video Generation (1)
Showing 1-25 of 25 articles
CRMArena / CRMArena-Pro
CRMArena is an AI benchmark for evaluating large language model agents on professional customer relationship management (CRM) tasks inside a realistic, schema-faithful Salesforce environment.
CRUXEval
CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation) is a benchmark designed to measure how well large language models can reason about, understand, and mentally execute short Python programs.
CodeContests
CodeContests is a competitive programming dataset created by Google DeepMind for training and evaluating machine learning models on algorithmic problem-solving tasks.
FeatureBench
FeatureBench is an execution-based benchmark for measuring how well LLM-powered coding agents handle complex, feature-oriented software development rather than bug fixing.
GSO (Global Software Optimization Bench)
GSO (Global Software Optimization), also styled as a benchmark of "Challenging Software Optimization Tasks for Evaluating SWE-Agents," is an AI benchmark that measures whether AI agents and language models can…
HumanEval
HumanEval is a benchmark for measuring whether a code-generating large language model can complete short Python functions so that they pass unit tests. It contains 164 hand-written tasks.
LiveCodeBench
LiveCodeBench is a holistic and contamination-free benchmark for evaluating large language models on code, first released in March 2024 by researchers at UC Berkeley, MIT, and Cornell led by Naman Jain.
MBPP
MBPP (Mostly Basic Python Problems) is a code generation benchmark of 974 crowd-sourced Python programming tasks designed to be solvable by entry-level programmers, introduced by Jacob Austin, Augustus Odena…
Multi-SWE-bench
Multi-SWE-bench is a multilingual benchmark for evaluating the ability of large language model based coding systems to resolve real-world software issues across seven programming languages.
MultiChallenge
MultiChallenge is an AI benchmark for evaluating large language models on realistic multi-turn conversations.
Pass@k
Pass@k is the standard metric for evaluating code generation models: it measures the probability that at least one of k generated candidate solutions passes all of a problem's unit tests.
Real-SWE
Real-SWE is a coding-agent benchmark published in September 2026 by Specific Labs, a San Francisco company that licenses operational data and source code from businesses and packages them as training and…
RepoBench
RepoBench is an AI benchmark for repository-level code auto-completion, introduced in the 2023 paper "RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems" by Tianyang Liu, Canwen Xu, and…
SWE-Atlas
SWE-Atlas is a benchmark for evaluating AI coding agents on professional software-engineering work that goes beyond fixing bugs and resolving issues.
SWE-Bench Pro
SWE-Bench Pro (stylized SWE-BENCH PRO) is a contamination-resistant benchmark, released by Scale AI in September 2025, that measures whether an AI coding agent can resolve long-horizon
SWE-Lancer
SWE-Lancer is a benchmark released by OpenAI in February 2025 that evaluates the ability of frontier large language models to perform real-world freelance software-engineering work.
SWE-bench Multilingual
SWE-bench Multilingual is an AI benchmark of 300 real-world software bug-fixing tasks drawn from 42 open-source repositories across nine programming languages
SWE-bench Multimodal
SWE-bench Multimodal (also written SWE-bench M) is a benchmark that measures whether autonomous software-engineering systems can resolve bugs in visual
SWE-bench Verified
SWE-bench Verified is a 500-problem, human-validated subset of the SWE-bench software engineering benchmark, released on August 13
SWE-rebench
SWE-rebench is a continuously refreshed, contamination-resistant AI benchmark and public leaderboard for evaluating AI agents on real-world software engineering tasks.
Spider 2.0
Spider 2.0 is a benchmark for evaluating large language models on real-world enterprise text-to-SQL workflows.
Terminal-Bench
Terminal-Bench is an open benchmark for evaluating AI agents on complex, real-world tasks performed through command-line terminal interfaces.
TheAgentCompany
TheAgentCompany is an AI benchmark that evaluates AI agents on long-horizon, economically valuable knowledge work inside a self-hosted simulation of a small software company.
Vibe Code Bench
Vibe Code Bench (VCB) is a benchmark that measures whether AI models can build a complete, deployable web application from nothing but a written specification.