Tokens per second
Tokens per second (TPS) is a key performance metric for measuring the speed of large language model (LLM) inference. It quantifies how many tokens a model can generate or process in one second.
Explore AI Benchmarks through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of AI Benchmarks.
Showing 241-268 of 268 articles
Tokens per second (TPS) is a key performance metric for measuring the speed of large language model (LLM) inference. It quantifies how many tokens a model can generate or process in one second.
ToxiGen is a large-scale, machine-generated dataset designed for adversarial and implicit hate speech detection.
TriviaQA is a large-scale reading comprehension and question answering dataset of over 650,000 question-answer-evidence triples, introduced in 2017 by Mandar Joshi, Eunsol Choi, Daniel S. Weld
TruthfulQA is a benchmark designed to measure whether large language models (LLMs) generate truthful answers to questions.
Vending-Bench is a family of benchmarks from Andon Labs that measures the long-term coherence of large language model-based AI agents by tasking them with running a simulated vending-machine business.
Vibe Code Bench (VCB) is a benchmark that measures whether AI models can build a complete, deployable web application from nothing but a written specification.
Video-MME (Video Multi-Modal Evaluation) is a benchmark for testing how well multimodal large language models (MLLMs) understand video, built from 900 manually selected videos totaling 254 hours and 2,700…
Video-MMMU (Video Multi-Modal Multi-disciplinary Understanding, sometimes written VideoMMMU) is a benchmark that measures whether Large Multimodal Models can acquire new knowledge from professional educational…
vimgolf-gym is an OpenAI Gym style customizable environment and benchmark built around VimGolf, the long running keystroke counting puzzle game for Vim.
Visual question answering models are AI systems that take an image and a natural language question about that image and return a natural language answer.
Visual question answering (VQA) is the task of producing a natural-language answer to a natural-language question about an image.
VisualWebArena (often abbreviated VWA) is a benchmark of 910 realistic, visually grounded web tasks for evaluating multimodal autonomous AI agents, released in January 2024 by researchers at Carnegie Mellon…
VulcanBench is an open-source benchmark for AI coding agents, created and run by Morgan Linton, co-founder and CTO of the apparel-sizing AI company Bold Metrics.
WISE (World Knowledge-Informed Semantic Evaluation) is an AI benchmark that tests whether a text-to-image model actually possesses and correctly applies real-world knowledge when it draws a scene
The Weapons of Mass Destruction Proxy (WMDP) is a publicly released benchmark and unlearning testbed for large language models, introduced in the paper "The WMDP Benchmark: Measuring and Reducing Malicious Use…
WebArena is a realistic, self-hosted web environment and benchmark designed for developing and evaluating autonomous AI agents that perform tasks on the web.
WebDev Arena is a live, community-driven leaderboard that ranks large language models on their ability to generate working web applications.
WebVoyager is an end-to-end web agent and its companion benchmark, introduced by Hongliang He and seven coauthors in a paper accepted to ACL 2024.
WeirdML is a benchmark for evaluating how well large language models can do hands on machine learning engineering.
WildBench is an automated evaluation framework for large language models (LLMs) introduced by the Allen Institute for AI (AI2) in 2024.
Windows Agent Arena (WAA) is a reproducible benchmark and evaluation environment created by Microsoft for testing multimodal computer-use agents on real Windows tasks at scale.
WindowsWorld is a process-aware benchmark for evaluating autonomous computer-use agents on professional workflows that cross several Windows applications.
WinoGrande is a large-scale benchmark for commonsense reasoning consisting of 44,000 binary fill-in-the-blank pronoun resolution problems, built to test whether language models genuinely understand commonsense…
The Winograd Schema Challenge (WSC) is a commonsense reasoning test in which a system must resolve an ambiguous pronoun in a short sentence where the correct answer flips when one or two words change
WritingBench is a comprehensive benchmark for evaluating the generative writing capabilities of large language models (LLMs) across diverse real-world writing tasks.
ZebraLogic is a benchmark for evaluating the logical reasoning capabilities of large language models (LLMs).
ZeroBench is a visual reasoning benchmark built to be effectively impossible for current frontier large multimodal models, which score 0.0% on its main questions.
τ-bench (Tau-bench), short for Tool-Agent-User Interaction Benchmark, is an AI benchmark that evaluates language agents' ability to complete complex tasks through realistic, multi-turn interactions with…