Tokens per second

Tokens per second (TPS) is a key performance metric for measuring the speed of large language model (LLM) inference. It quantifies how many tokens a model can generate or process in one second. Higher TPS values indicate faster generation, which translates to shorter wait times for users and lower costs for providers. TPS is one of the most commonly cited metrics when evaluating and comparing LLM serving infrastructure, and it plays a central role in benchmarks published by organizations such as Artificial Analysis, NVIDIA, and cloud service providers.

As LLMs have grown in capability and deployment scale, TPS has become a critical factor in hardware procurement decisions, model selection, and infrastructure design. The metric is closely related to other inference performance measures including time to first token (TTFT), inter-token latency (ITL), and throughput.

Definition and measurement

At its most basic, tokens per second measures the rate at which tokens are produced during inference. However, the precise definition varies depending on what phase of inference is being measured and whether the metric refers to a single request or the entire system.

Output TPS (per-request)

Output TPS measures how many tokens a model generates per second for a single request, excluding the initial prompt processing time. This is the metric most relevant to the end-user experience during streaming, as it determines how quickly text appears on screen after generation begins.^[1][9]

Output TPS = (number of output tokens) / (time spent generating output tokens)

Total TPS (per-request)

Total TPS accounts for the complete request lifecycle, including both prompt processing (prefill) and token generation (decode):

Total TPS = (number of output tokens) / (total request time)

Total TPS is always lower than output TPS because it includes the prefill phase, during which no output tokens are produced.^[1]

System throughput TPS

System throughput TPS measures the total number of tokens generated per second across all concurrent requests being served by the system:

System throughput TPS = (total output tokens across all requests) / (time period)

This metric is most relevant for infrastructure planning and cost analysis, as it determines how many requests a deployment can handle simultaneously.^[1]

Scroll sideways for more →

Time to first token (TTFT)

Time to first token (TTFT) measures the time between sending a request and receiving the first output token.^[1][10] It is distinct from TPS but closely related, as both contribute to the overall user experience.

TTFT = time when first token is received - time when request is sent

TTFT includes several components:

Scroll sideways for more →

TTFT is particularly important for interactive applications like chatbots and AI coding assistants, where users expect near-instant acknowledgment that the model has started generating. A TTFT of under 500 milliseconds is generally considered acceptable for real-time conversational applications, while batch processing pipelines are less sensitive to TTFT.

TTFT vs. output speed

TTFT and output TPS measure different phases of inference and are affected by different factors:

Scroll sideways for more →

For reasoning models that produce internal chain-of-thought tokens before the visible response, TTFT may include the time to generate reasoning tokens, which can be substantial (seconds to minutes for complex queries).

Factors affecting TPS

Many factors influence tokens per second, spanning hardware, model architecture, software optimization, and serving configuration.

Model size

Model size (measured in parameters) is the most fundamental determinant of TPS. Larger models require more computation per token and more memory bandwidth to load weights, directly reducing generation speed.

Scroll sideways for more →

These are rough estimates and vary significantly depending on hardware, software stack, quantization, and batch size.

Quantization

Quantization reduces the precision of model weights from higher-precision formats (FP32, FP16, BF16) to lower-precision formats (INT8, INT4, FP8). This reduces memory usage and increases TPS, often with minimal impact on output quality.

Scroll sideways for more →

Quantization improves TPS primarily by reducing memory bandwidth requirements. During the decode phase, LLM inference is typically memory-bandwidth bound: the bottleneck is loading model weights from GPU memory, not performing computations.^[1] Smaller weights mean less data to transfer, which translates directly to higher TPS.

Notable quantization methods include GPTQ (post-training quantization using Hessian-based optimization), AWQ (activation-aware weight quantization), and GGUF (the format used by llama.cpp for CPU and mixed CPU-GPU inference).^[7][8]

Hardware

The choice of hardware profoundly affects TPS. The two most important hardware characteristics for LLM inference are memory bandwidth and compute throughput.

Scroll sideways for more →

For the memory-bandwidth-bound decode phase, memory bandwidth is the primary determinant of TPS. The NVIDIA H100 achieves roughly 2.8 times the inference throughput of the A100, closely tracking its ~1.7x advantage in memory bandwidth combined with architectural improvements in the Transformer Engine and FP8 support.^[1]

Batch size

Batch size (the number of requests processed simultaneously) significantly affects both per-request TPS and system throughput TPS, but in opposite directions.

Scroll sideways for more →

At small batch sizes, inference is memory-bandwidth bound, and increasing the batch size improves total throughput without proportionally increasing latency. At large batch sizes, inference becomes compute bound, and further increases in batch size begin to slow down individual requests.

Continuous batching (also called iteration-level batching), used by serving frameworks like vLLM and TensorRT-LLM, dynamically adds new requests to the batch as existing requests complete. This maintains high throughput without the latency penalty of waiting for a full batch to accumulate.

Software optimization

The inference software stack has a large impact on TPS. Key optimizations include:

Scroll sideways for more →

Context length

The length of the input prompt (context) affects both TTFT and, to a lesser extent, output TPS. Longer prompts require more time for the prefill phase and produce larger KV caches, which increase memory pressure during the decode phase.

Scroll sideways for more →

Benchmarking methodology

Accurate TPS benchmarking requires careful methodology to produce meaningful, reproducible results.

Key considerations

1. Standardized tokenization. Different models use different tokenizers, so the same text may produce different token counts depending on the model. Artificial Analysis standardizes all measurements to OpenAI GPT-4 tokens (o200k_base tokenizer) to allow fair cross-model comparisons.^[2] Other benchmarks may use each model's native tokenizer, which can make comparisons misleading.

2. Warm-up runs. Initial requests may be slower due to model loading, kernel compilation (for JIT-compiled frameworks), and cache population. Benchmarks should include warm-up requests that are excluded from measurements.

3. Statistical reporting. TPS can vary between requests due to server load, network conditions, and other factors. Robust benchmarks report statistical aggregates. Artificial Analysis reports the median (P50) over the past 72 hours to reflect sustained performance.^[3] Other benchmarks may report P50, P90, P95, or P99 latencies.

4. Fixed prompt and output lengths. For comparable results, benchmarks should use standardized prompt lengths and target output lengths. Artificial Analysis uses several standardized workloads including short prompts and 100-token outputs.^[3]

5. Concurrent requests. Benchmarks should specify the level of concurrency. Single-request benchmarks measure per-user speed; high-concurrency benchmarks measure system throughput.

Artificial Analysis

Artificial Analysis is the most widely referenced independent benchmarking platform for LLM API performance. Founded in 2023, it continuously benchmarks over 300 models and 500 API endpoints across providers including OpenAI, Anthropic, Google, Mistral, and numerous open-model hosting providers.^[2]

Key aspects of their methodology:

• All token measurements use OpenAI's o200k_base tokenizer for cross-model fairness.^[2]
• Performance metrics are reported as median (P50) values over rolling 72-hour windows.^[3]
• Testing servers are located in Google Cloud's us-central1-a zone.^[3]
• For reasoning models, TTFT measures the time to the first reasoning token.^[3]
• They test multiple prompt lengths including a 100K-token workload tested weekly.^[3]

As of early 2026, some of the fastest models on the Artificial Analysis leaderboard include Mercury Coder (835+ tokens/second), NVIDIA Nemotron models (400+ tokens/second), and various optimized 7-8B parameter models (300+ tokens/second).

NVIDIA benchmarking tools

NVIDIA provides benchmarking tools through its NIM (NVIDIA Inference Microservices) platform and TensorRT-LLM library. Their AIPerf benchmark measures throughput, TTFT, and inter-token latency under controlled conditions.^[1] NVIDIA's benchmarks are particularly useful for evaluating the impact of hardware choices and optimization settings.

Other benchmarking frameworks

Scroll sideways for more →

TPS across deployment scenarios

TPS requirements and achievable values differ dramatically across deployment scenarios.

Scroll sideways for more →

TPS and user experience

The relationship between TPS and user experience follows a pattern of diminishing returns. Research and user studies suggest the following thresholds:

Scroll sideways for more →

The average adult reads at approximately 200-300 words per minute, which corresponds to roughly 5-8 words per second or about 7-10 tokens per second (given average tokenization ratios). Output speeds of 30+ TPS therefore significantly exceed human reading speed, and further improvements primarily benefit batch processing, programmatic use cases, and TTFT-sensitive applications rather than direct user perception.

TPS and cost

TPS is directly related to the cost of serving LLM requests. Higher TPS means each GPU-second produces more tokens, reducing the cost per token.

Cost per million tokens (approx.) = (GPU cost per second * 1,000,000) / (system throughput TPS)

For example, if an H100 server costs $3 per hour ($0.000833 per second) and achieves a system throughput of 1,000 TPS, the cost per million output tokens is approximately $0.83. Improving throughput to 2,000 TPS would halve this cost to approximately $0.42 per million tokens.

This economic relationship drives the intense industry focus on inference optimization, as even small improvements in TPS translate to significant cost savings at scale.

Historical trends

TPS for frontier models has improved steadily due to advances in hardware, software optimization, and model architecture.

Scroll sideways for more →

Common misconceptions

Scroll sideways for more →

References

1. NVIDIA. "LLM Inference Benchmarking: Fundamental Concepts." NVIDIA Technical Blog, 2025. https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts/ ↩
2. Artificial Analysis. "Language Model Benchmarking Methodology." https://artificialanalysis.ai/methodology ↩
3. Artificial Analysis. "Language Model API Performance Benchmarking." https://artificialanalysis.ai/methodology/performance-benchmarking ↩
4. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023). "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP 2023. ↩
5. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Re, C. (2022). "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." NeurIPS 2022. ↩
6. Leviathan, Y., Kalman, M., & Matias, Y. (2023). "Fast Inference from Transformers via Speculative Decoding." ICML 2023. ↩
7. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." ICLR 2023. ↩
8. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., & Han, S. (2024). "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." MLSys 2024. ↩
9. BentoML. "Key metrics for LLM inference." LLM Inference Handbook. https://bentoml.com/llm/inference-optimization/llm-inference-metrics ↩
10. Anyscale. "Understand LLM latency and throughput metrics." Anyscale Documentation. https://docs.anyscale.com/llm/serving/benchmarking/metrics ↩