LLM Comparisons: Difference between revisions
| Line 3,615: | Line 3,615: | ||
| Model Providers | | Model Providers | ||
|} | |} | ||
== | ==Terms== | ||
*'''Artificial Analysis Quality Index''': Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details. | |||
*''Context window''': Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model). | |||
*''Output Speed''': Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming). | |||
*'''Latency''': Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion. | |||
*''Price''': Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio). | |||
*''Output Price''': Price per token generated by the model (received from the API), represented as USD per million Tokens. | |||
*''Input Price''': Price per token included in the request/message sent to the API, represented as USD per million Tokens. | |||
*''Time period''': Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests. | |||
==References== | ==References== | ||
Revision as of 23:51, 12 January 2025
Concised Comparison
| Model | Creator | Context Window | Quality Index | Normalized avg | Blended | USD/1M Tokens | Median Tokens/s | Median First Chunk (s) | Further Analysis |
|---|---|---|---|---|---|---|---|---|---|
| o1-preview | OpenAI logo | 128k | 86 | – | $27.56 | – | 144.5 | 21.12 | Model Providers |
| o1-mini | OpenAI logo | 128k | 84 | – | $5.25 | – | 213.2 | 11.64 | Model Providers |
| GPT-4o (Aug '24) | OpenAI logo | 128k | 78 | – | $4.38 | – | 85.6 | 0.65 | Model Providers |
| GPT-4o (May '24) | OpenAI logo | 128k | 78 | – | $7.50 | – | 106.6 | 0.65 | Model Providers |
| GPT-4o mini | OpenAI logo | 128k | 73 | – | $0.26 | – | 121.8 | 0.65 | Model Providers |
| GPT-4o (Nov '24) | OpenAI logo | 128k | 73 | – | $4.38 | – | 115.7 | 0.38 | Model Providers |
| GPT-4o mini Realtime (Dec '24) | OpenAI logo | 128k | – | – | $0.00 | – | – | – | Model Providers |
| GPT-4o Realtime (Dec '24) | OpenAI logo | 128k | – | – | $0.00 | – | – | – | Model Providers |
| Llama 3.3 70B | Meta logo | 128k | 74 | – | $0.69 | – | 71.3 | 0.50 | Model Providers |
| Llama 3.1 405B | Meta logo | 128k | 74 | – | $3.50 | – | 30.2 | 0.72 | Model Providers |
| Llama 3.1 70B | Meta logo | 128k | 68 | – | $0.72 | – | 72.7 | 0.44 | Model Providers |
| Llama 3.2 90B (Vision) | Meta logo | 128k | 68 | – | $0.81 | – | 48.5 | 0.34 | Model Providers |
| Llama 3.2 11B (Vision) | Meta logo | 128k | 54 | – | $0.18 | – | 131.0 | 0.29 | Model Providers |
| Llama 3.1 8B | Meta logo | 128k | 54 | – | $0.10 | – | 184.7 | 0.34 | Model Providers |
| Llama 3.2 3B | Meta logo | 128k | 49 | – | $0.06 | – | 202.1 | 0.38 | Model Providers |
| Llama 3.2 1B | Meta logo | 128k | 26 | – | $0.04 | – | 468.9 | 0.38 | Model Providers |
| Gemini 2.0 Flash (exp) | Google logo | 1m | 82 | – | $0.00 | – | 169.0 | 0.48 | Model Providers |
| Gemini 1.5 Pro (Sep) | Google logo | 2m | 80 | – | $2.19 | – | 60.9 | 0.74 | Model Providers |
| Gemini 1.5 Flash (Sep) | Google logo | 1m | 72 | – | $0.13 | – | 188.6 | 0.25 | Model Providers |
| Gemma 2 27B | Google logo | 8k | 61 | – | $0.26 | – | 58.9 | 0.51 | Model Providers |
| Gemma 2 9B | Google logo | 8k | 55 | – | $0.12 | – | 168.9 | 0.37 | Model Providers |
| Gemini 1.5 Flash (May) | Google logo | 1m | – | – | $0.13 | – | 310.4 | 0.29 | Model Providers |
| Gemini Experimental (Nov) | Google logo | 2m | – | – | $0.00 | – | 53.9 | 1.11 | Model Providers |
| Gemini 1.5 Pro (May) | Google logo | 2m | – | – | $2.19 | – | 66.9 | 0.50 | Model Providers |
| Gemini 1.5 Flash-8B | Google logo | 1m | – | – | $0.07 | – | 278.8 | 0.39 | Model Providers |
| Claude 3.5 Sonnet (Oct) | Anthropic logo | 200k | 80 | – | $6.00 | – | 71.9 | 0.98 | Model Providers |
| Claude 3.5 Sonnet (June) | Anthropic logo | 200k | 76 | – | $6.00 | – | 61.4 | 0.87 | Model Providers |
| Claude 3 Opus | Anthropic logo | 200k | 70 | – | $30.00 | – | 25.9 | 2.00 | Model Providers |
| Claude 3.5 Haiku | Anthropic logo | 200k | 68 | – | $1.60 | – | 65.1 | 0.71 | Model Providers |
| Claude 3 Haiku | Anthropic logo | 200k | 55 | – | $0.50 | – | 122.7 | 0.72 | Model Providers |
| Pixtral Large | Mistral logo | 128k | 74 | – | $3.00 | – | 36.7 | 0.39 | Model Providers |
| Mistral Large 2 (Jul '24) | Mistral logo | 128k | 74 | – | $3.00 | – | 31.2 | 0.50 | Model Providers |
| Mistral Large 2 (Nov '24) | Mistral logo | 128k | 74 | – | $3.00 | – | 37.4 | 0.52 | Model Providers |
| Mistral Small (Sep '24) | Mistral logo | 33k | 61 | – | $0.30 | – | 61.5 | 0.32 | Model Providers |
| Mixtral 8x22B | Mistral logo | 65k | 61 | – | $1.20 | – | 85.5 | 0.57 | Model Providers |
| Pixtral 12B | Mistral logo | 128k | 56 | – | $0.13 | – | 70.5 | 0.37 | Model Providers |
| Ministral 8B | Mistral logo | 128k | 56 | – | $0.10 | – | 135.8 | 0.29 | Model Providers |
| Mistral NeMo | Mistral logo | 128k | 54 | – | $0.09 | – | 123.7 | 0.48 | Model Providers |
| Ministral 3B | Mistral logo | 128k | 53 | – | $0.04 | – | 168.1 | 0.29 | Model Providers |
| Mixtral 8x7B | Mistral logo | 33k | 41 | – | $0.50 | – | 110.1 | 0.36 | Model Providers |
| Codestral-Mamba | Mistral logo | 256k | 33 | – | $0.25 | – | 95.8 | 0.44 | Model Providers |
| Command-R+ | Cohere logo | 128k | 55 | – | $5.19 | – | 50.7 | 0.47 | Model Providers |
| Command-R+ (Apr '24) | Cohere logo | 128k | 45 | – | $6.00 | – | 49.3 | 0.51 | Model Providers |
| Command-R (Mar '24) | Cohere logo | 128k | 36 | – | $0.75 | – | 108.3 | 0.36 | Model Providers |
| Aya Expanse 8B | Cohere logo | 8k | – | – | $0.75 | – | 165.6 | 0.16 | Model Providers |
| Command-R | Cohere logo | 128k | – | – | $0.51 | – | 111.7 | 0.32 | Model Providers |
| Aya Expanse 32B | Cohere logo | 128k | – | – | $0.75 | – | 120.3 | 0.18 | Model Providers |
| Sonar 3.1 Small | Perplexity logo | 127k | – | – | $0.20 | – | 203.8 | 0.35 | Model Providers |
| Sonar 3.1 Large | Perplexity logo | 127k | – | – | $1.00 | – | 57.8 | 0.31 | Model Providers |
| Grok Beta | xAI logo | 128k | 72 | – | $7.50 | – | 66.5 | 0.42 | Model Providers |
| Nova Pro | Amazon logo | 300k | 75 | – | $1.40 | – | 91.1 | 0.38 | Model Providers |
| Nova Lite | Amazon logo | 300k | 70 | – | $0.10 | – | 147.7 | 0.34 | Model Providers |
| Nova Micro | Amazon logo | 130k | 66 | – | $0.06 | – | 195.3 | 0.33 | Model Providers |
| Phi-4 | Microsoft Azure logo | 16k | 77 | – | $0.09 | – | 85.1 | 0.21 | Model Providers |
| Phi-3 Mini | Microsoft Azure logo | 4k | – | – | $0.00 | – | – | – | Model Providers |
| Phi-3 Medium 14B | Microsoft Azure logo | 128k | – | – | $0.30 | – | 50.7 | 0.43 | Model Providers |
| Solar Mini | Upstage logo | 4k | 47 | – | $0.15 | – | 89.3 | 1.12 | Model Providers |
| DBRX | Databricks logo | 33k | 46 | – | $1.16 | – | 74.2 | 0.43 | Model Providers |
| Llama 3.1 Nemotron 70B | NVIDIA logo | 128k | 72 | – | $0.27 | – | 48.3 | 0.57 | Model Providers |
| Reka Flash | Reka AI logo | 128k | 59 | – | $0.35 | – | – | – | Model Providers |
| Reka Core | Reka AI logo | 128k | 58 | – | $2.00 | – | – | – | Model Providers |
| Reka Flash (Feb '24) | Reka AI logo | 128k | 46 | – | $0.35 | – | – | – | Model Providers |
| Reka Edge | Reka AI logo | 128k | 31 | – | $0.10 | – | – | – | Model Providers |
| Jamba 1.5 Large | AI21 Labs logo | 256k | 64 | – | $3.50 | – | 50.8 | 0.71 | Model Providers |
| Jamba 1.5 Mini | AI21 Labs logo | 256k | – | – | $0.25 | – | 83.5 | 0.48 | Model Providers |
| DeepSeek V3 | DeepSeek logo | 128k | 80 | – | $0.90 | – | 21.1 | 0.95 | Model Providers |
| DeepSeek-V2.5 (Dec '24) | DeepSeek logo | 128k | 72 | – | $0.17 | – | 63.8 | 1.15 | Model Providers |
| DeepSeek-Coder-V2 | DeepSeek logo | 128k | 71 | – | $0.17 | – | 63.8 | 1.12 | Model Providers |
| DeepSeek-V2.5 | DeepSeek logo | 128k | – | – | $1.09 | – | 7.6 | 0.77 | Model Providers |
| DeepSeek-V2 | DeepSeek logo | 128k | – | – | $0.17 | – | – | – | Model Providers |
| Arctic | Snowflake logo | 4k | 51 | – | $0.00 | – | – | – | Model Providers |
| Qwen2.5 72B | Alibaba logo | 131k | 77 | – | $0.40 | – | 67.6 | 0.53 | Model Providers |
| Qwen2.5 Coder 32B | Alibaba logo | 131k | 72 | – | $0.80 | – | 84.1 | 0.38 | Model Providers |
| Qwen2 72B | Alibaba logo | 131k | 72 | – | $0.63 | – | 45.7 | 0.30 | Model Providers |
| QwQ 32B-Preview | Alibaba logo | 33k | 46 | – | $0.26 | – | 66.9 | 0.40 | Model Providers |
| Yi-Large | 01.AI logo | 32k | 61 | – | $3.00 | – | 67.5 | 0.47 | Model Providers |
| GPT-4 Turbo | OpenAI logo | 128k | 75 | – | $15.00 | – | 43.8 | 1.19 | Model Providers |
| GPT-4 | OpenAI logo | 8k | – | – | $37.50 | – | 30.5 | 0.73 | Model Providers |
| Llama 3 70B | Meta logo | 8k | 48 | – | $0.89 | – | 48.9 | 0.38 | Model Providers |
| Llama 3 8B | Meta logo | 8k | 45 | – | $0.15 | – | 118.9 | 0.34 | Model Providers |
| Llama 2 Chat 70B | Meta logo | 4k | – | – | $1.85 | – | – | – | Model Providers |
| Llama 2 Chat 13B | Meta logo | 4k | – | – | $0.00 | – | – | – | Model Providers |
| Llama 2 Chat 7B | Meta logo | 4k | – | – | $0.33 | – | 123.8 | 0.37 | Model Providers |
| Gemini 1.0 Pro | Google logo | 33k | – | – | $0.75 | – | 103.0 | 1.28 | Model Providers |
| Claude 3 Sonnet | Anthropic logo | 200k | 57 | – | $6.00 | – | 67.9 | 0.74 | Model Providers |
| Claude 2.1 | Anthropic logo | 200k | – | – | $12.00 | – | 20.3 | 1.41 | Model Providers |
| Claude 2.0 | Anthropic logo | 100k | – | – | $12.00 | – | 29.9 | 0.81 | Model Providers |
| Mistral Small (Feb '24) | Mistral logo | 33k | 59 | – | $1.50 | – | 53.7 | 0.38 | Model Providers |
| Mistral Large (Feb '24) | Mistral logo | 33k | 56 | – | $6.00 | – | 38.9 | 0.43 | Model Providers |
| Mistral 7B | Mistral logo | 8k | 28 | – | $0.16 | – | 113.1 | 0.26 | Model Providers |
| Mistral Medium | Mistral logo | 33k | – | – | $4.09 | – | 44.5 | 0.38 | Model Providers |
| Codestral | Mistral logo | 33k | – | – | $0.30 | – | 84.9 | 0.28 | Model Providers |
| OpenChat 3.5 | OpenChat logo | 8k | 44 | – | $0.06 | – | 73.4 | 0.30 | Model Providers |
| Jamba Instruct | AI21 Labs logo | 256k | – | – | $0.55 | – | 77.3 | 0.52 | Model Providers |
Detailed Comparison
| Model | Creator | License | Context Window | Quality Index | Normalized avg | Chatbot Arena | MMLU | GPQA | MATH-500 | HumanEval | Blended | USD/1M Tokens | Input Price | Output Price | Median Tokens/s | P5 Tokens/s | P25 Tokens/s | P75 Tokens/s | P95 Tokens/s | Median First Chunk (s) | P5 First Chunk (s) | P25 First Chunk (s) | P75 First Chunk (s) | P95 First Chunk (s) | Further Analysis |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| o1-preview | OpenAI logo | Proprietary | 128k | 86 | 1334 | 0.91 | 0.67 | 0.92 | 0.96 | $27.56 | $15.75 | $63.00 | 143.8 | 68.9 | 121.6 | 164.6 | 179.6 | 21.28 | 13.40 | 17.04 | 27.80 | 46.49 | – | – | Model Providers |
| o1-mini | OpenAI logo | Proprietary | 128k | 84 | 1308 | 0.85 | 0.58 | 0.95 | 0.97 | $5.25 | $3.00 | $12.00 | 213.6 | 84.0 | 154.8 | 238.0 | 299.4 | 11.75 | 2.44 | 9.40 | 14.43 | 24.03 | – | – | Model Providers |
| GPT-4o (Aug '24) | OpenAI logo | Proprietary | 128k | 78 | 1337 | 0.89 | 0.51 | 0.80 | 0.93 | – | – | $4.38 | $2.50 | $10.00 | 85.6 | 40.3 | 61.5 | 109.3 | 143.6 | 0.66 | 0.33 | 0.43 | 0.91 | 1.92 | Model Providers |
| GPT-4o (May '24) | OpenAI logo | Proprietary | 128k | 78 | 1285 | 0.87 | 0.51 | 0.79 | 0.93 | – | – | $7.50 | $5.00 | $15.00 | 106.8 | 53.2 | 82.2 | 126.8 | 142.5 | 0.65 | 0.32 | 0.43 | 0.73 | 1.22 | Model Providers |
| GPT-4o mini | OpenAI logo | Proprietary | 128k | 73 | 1273 | 0.82 | 0.44 | 0.79 | 0.88 | – | – | $0.26 | $0.15 | $0.60 | 121.8 | 50.7 | 74.1 | 179.4 | 206.5 | 0.65 | 0.30 | 0.39 | 0.77 | 0.92 | Model Providers |
| GPT-4o (Nov '24) | OpenAI logo | Proprietary | 128k | 73 | 1361 | 0.86 | 0.39 | 0.74 | 0.93 | – | – | $4.38 | $2.50 | $10.00 | 115.1 | 71.7 | 95.0 | 140.0 | 165.5 | 0.38 | 0.27 | 0.32 | 0.52 | 0.75 | Model Providers |
| GPT-4o mini Realtime (Dec '24) | OpenAI logo | Proprietary | 128k | – | – | – | – | – | – | – | – | $0.00 | $0.00 | $0.00 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| GPT-4o Realtime (Dec '24) | OpenAI logo | Proprietary | 128k | – | – | – | – | – | – | – | – | $0.00 | $0.00 | $0.00 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| Llama 3.3 70B | Meta logo | Open | 128k | 74 | – | 0.86 | 0.49 | 0.76 | 0.86 | – | – | $0.67 | $0.59 | $0.73 | 67.2 | 23.6 | 31.2 | 275.7 | 2046.5 | 0.51 | 0.23 | 0.36 | 0.72 | 1.48 | Model Providers |
| Llama 3.1 405B | Meta logo | Open | 128k | 74 | 1266 | 0.87 | 0.50 | 0.70 | 0.87 | – | – | $3.50 | $3.50 | $3.50 | 30.2 | 12.2 | 21.8 | 66.3 | 160.3 | 0.72 | 0.37 | 0.52 | 0.89 | 2.02 | Model Providers |
| Llama 3.1 70B | Meta logo | Open | 128k | 68 | 1249 | 0.84 | 0.43 | 0.64 | 0.80 | – | – | $0.70 | $0.60 | $0.80 | 72.6 | 27.9 | 42.2 | 218.5 | 1971.9 | 0.44 | 0.24 | 0.34 | 0.60 | 1.00 | Model Providers |
| Llama 3.2 90B (Vision) | Meta logo | Open | 128k | 68 | – | 0.84 | 0.42 | 0.65 | 0.80 | – | – | $0.81 | $0.81 | $0.81 | 48.6 | 30.4 | 35.9 | 66.6 | 272.3 | 0.34 | 0.18 | 0.26 | 0.47 | 0.61 | Model Providers |
| Llama 3.2 11B (Vision) | Meta logo | Open | 128k | 54 | – | 0.71 | 0.25 | 0.50 | 0.69 | – | – | $0.18 | $0.18 | $0.18 | 131.0 | 35.9 | 118.2 | 165.4 | 752.4 | 0.29 | 0.17 | 0.23 | 0.36 | 0.56 | Model Providers |
| Llama 3.1 8B | Meta logo | Open | 128k | 54 | 1172 | 0.71 | 0.26 | 0.50 | 0.68 | – | – | $0.10 | $0.10 | $0.10 | 182.8 | 49.5 | 118.8 | 468.1 | 2161.5 | 0.34 | 0.17 | 0.27 | 0.47 | 0.73 | Model Providers |
| Llama 3.2 3B | Meta logo | Open | 128k | 49 | 1103 | 0.64 | 0.21 | 0.50 | 0.60 | – | – | $0.06 | $0.06 | $0.06 | 202.2 | 42.4 | 144.0 | 543.6 | 1623.1 | 0.38 | 0.15 | 0.26 | 0.49 | 0.93 | Model Providers |
| Llama 3.2 1B | Meta logo | Open | 128k | 26 | 1054 | 0.35 | 0.14 | 0.13 | 0.40 | – | – | $0.04 | $0.04 | $0.04 | 315.4 | 179.4 | 261.4 | 2053.8 | 3394.4 | 0.44 | 0.20 | 0.30 | 0.51 | 0.74 | Model Providers |
| Gemini 2.0 Flash (exp) | Google logo | Proprietary | 1m | 82 | – | 0.87 | 0.59 | 0.90 | 0.91 | – | – | $0.00 | $0.00 | $0.00 | 168.8 | 162.4 | 165.5 | 171.4 | 174.6 | 0.48 | 0.41 | 0.45 | 0.53 | 0.73 | Model Providers |
| Gemini 1.5 Pro (Sep) | Google logo | Proprietary | 2m | 80 | 1301 | 0.86 | 0.59 | 0.88 | 0.88 | – | – | $2.19 | $1.25 | $5.00 | 60.9 | 55.3 | 59.0 | 63.7 | 66.1 | 0.74 | 0.36 | 0.39 | 0.80 | 0.91 | Model Providers |
| Gemini 1.5 Flash (Sep) | Google logo | Proprietary | 1m | 72 | 1271 | 0.75 | 0.45 | 0.83 | 0.83 | – | – | $0.13 | $0.07 | $0.30 | 188.5 | 172.8 | 182.0 | 194.7 | 204.0 | 0.25 | 0.19 | 0.21 | 0.44 | 0.49 | Model Providers |
| Gemma 2 27B | Google logo | Open | 8k | 61 | 1219 | 0.77 | 0.39 | 0.54 | 0.76 | – | – | $0.26 | $0.17 | $0.51 | 58.7 | 32.0 | 46.3 | 67.8 | 74.4 | 0.51 | 0.19 | 0.32 | 1.53 | 5.94 | Model Providers |
| Gemma 2 9B | Google logo | Open | 8k | 55 | 1190 | 0.73 | 0.31 | 0.50 | 0.64 | – | – | $0.12 | $0.12 | $0.15 | 168.9 | 53.4 | 114.8 | 187.3 | 673.7 | 0.37 | 0.22 | 0.27 | 0.50 | 0.65 | Model Providers |
| Gemini 1.5 Flash (May) | Google logo | Proprietary | 1m | – | 1227 | 0.79 | 0.39 | 0.55 | – | – | – | $0.13 | $0.07 | $0.30 | 310.0 | 276.8 | 297.5 | 325.0 | 350.4 | 0.30 | 0.23 | 0.27 | 0.33 | 0.39 | Model Providers |
| Gemini Experimental (Nov) | Google logo | Proprietary | 2m | – | 1365 | – | – | – | – | – | – | $0.00 | $0.00 | $0.00 | 53.8 | 51.1 | 52.6 | 55.1 | 56.4 | 1.12 | 0.78 | 0.90 | 1.82 | 3.31 | Model Providers |
| Gemini 1.5 Pro (May) | Google logo | Proprietary | 2m | – | 1260 | 0.86 | 0.46 | 0.66 | – | – | – | $2.19 | $1.25 | $5.00 | 66.9 | 62.7 | 64.6 | 68.4 | 70.2 | 0.50 | 0.38 | 0.42 | 0.81 | 0.88 | Model Providers |
| Gemini 1.5 Flash-8B | Google logo | Proprietary | 1m | – | 1211 | 0.75 | 0.30 | 0.70 | – | – | – | $0.07 | $0.04 | $0.15 | 279.0 | 226.9 | 260.1 | 288.8 | 300.0 | 0.39 | 0.27 | 0.35 | 0.46 | 0.59 | Model Providers |
| Claude 3.5 Sonnet (Oct) | Anthropic logo | Proprietary | 200k | 80 | 1282 | 0.89 | 0.58 | 0.76 | 0.96 | – | – | $6.00 | $3.00 | $15.00 | 71.8 | 37.6 | 44.8 | 78.0 | 89.6 | 0.98 | 0.68 | 0.78 | 1.36 | 2.23 | Model Providers |
| Claude 3.5 Sonnet (June) | Anthropic logo | Proprietary | 200k | 76 | 1268 | 0.88 | 0.56 | 0.71 | 0.90 | – | – | $6.00 | $3.00 | $15.00 | 61.4 | 41.6 | 49.9 | 78.9 | 91.0 | 0.87 | 0.68 | 0.75 | 1.06 | 1.45 | Model Providers |
| Claude 3 Opus | Anthropic logo | Proprietary | 200k | 70 | 1248 | 0.84 | 0.50 | 0.64 | 0.83 | – | – | $30.00 | $15.00 | $75.00 | 25.9 | 20.8 | 24.0 | 28.2 | 30.2 | 2.01 | 1.31 | 1.54 | 3.09 | 3.78 | Model Providers |
| Claude 3.5 Haiku | Anthropic logo | Proprietary | 200k | 68 | – | 0.81 | 0.37 | 0.67 | 0.87 | – | – | $1.60 | $0.80 | $4.00 | 65.1 | 51.1 | 58.6 | 75.4 | 105.1 | 0.71 | 0.54 | 0.64 | 0.93 | 1.20 | Model Providers |
| Claude 3 Haiku | Anthropic logo | Proprietary | 200k | 55 | 1179 | 0.71 | 0.33 | 0.39 | 0.77 | – | – | $0.50 | $0.25 | $1.25 | 122.6 | 97.6 | 112.0 | 134.7 | 152.7 | 0.72 | 0.35 | 0.43 | 0.79 | 1.06 | Model Providers |
| Pixtral Large | Mistral logo | Open | 128k | 74 | – | 0.85 | 0.52 | 0.71 | 0.88 | – | – | $3.00 | $2.00 | $6.00 | 36.6 | 18.9 | 33.6 | 39.3 | 41.1 | 0.40 | 0.33 | 0.34 | 0.51 | 2.22 | Model Providers |
| Mistral Large 2 (Jul '24) | Mistral logo | Open | 128k | 74 | 1251 | 0.85 | 0.48 | 0.71 | 0.91 | – | – | $3.00 | $2.00 | $6.00 | 31.2 | 25.8 | 29.2 | 33.8 | 35.4 | 0.50 | 0.38 | 0.45 | 0.55 | 0.89 | Model Providers |
| Mistral Large 2 (Nov '24) | Mistral logo | Open | 128k | 74 | – | 0.85 | 0.47 | 0.72 | 0.90 | – | – | $3.00 | $2.00 | $6.00 | 37.4 | 32.2 | 36.0 | 41.0 | 66.3 | 0.52 | 0.33 | 0.45 | 0.56 | 0.73 | Model Providers |
| Mistral Small (Sep '24) | Mistral logo | Open | 33k | 61 | – | 0.74 | 0.34 | 0.56 | 0.81 | – | – | $0.30 | $0.20 | $0.60 | 62.0 | 50.6 | 57.0 | 68.2 | 77.1 | 0.32 | 0.23 | 0.25 | 0.52 | 1.50 | Model Providers |
| Mixtral 8x22B | Mistral logo | Open | 65k | 61 | 1148 | 0.76 | 0.37 | 0.56 | 0.74 | – | – | $1.20 | $1.20 | $1.20 | 85.6 | 57.2 | 69.9 | 96.6 | 102.5 | 0.57 | 0.26 | 0.34 | 0.65 | 0.87 | Model Providers |
| Pixtral 12B | Mistral logo | Open | 128k | 56 | – | 0.69 | 0.30 | 0.46 | 0.79 | – | – | $0.13 | $0.13 | $0.13 | 70.5 | 61.6 | 66.8 | 77.3 | 81.0 | 0.37 | 0.26 | 0.31 | 0.48 | 0.74 | Model Providers |
| Ministral 8B | Mistral logo | Open | 128k | 56 | 1183 | 0.59 | 0.30 | 0.57 | 0.79 | – | – | $0.10 | $0.10 | $0.10 | 135.9 | 121.0 | 131.3 | 138.5 | 141.5 | 0.29 | 0.23 | 0.25 | 0.34 | 0.60 | Model Providers |
| Mistral NeMo | Mistral logo | Open | 128k | 54 | – | 0.66 | 0.33 | 0.44 | 0.71 | – | – | $0.09 | $0.06 | $0.14 | 124.0 | 51.8 | 76.5 | 158.8 | 163.6 | 0.48 | 0.18 | 0.25 | 0.55 | 0.71 | Model Providers |
| Ministral 3B | Mistral logo | Proprietary | 128k | 53 | – | 0.58 | 0.26 | 0.54 | 0.74 | – | – | $0.04 | $0.04 | $0.04 | 168.3 | 147.8 | 163.9 | 173.1 | 177.7 | 0.29 | 0.22 | 0.26 | 0.34 | 0.55 | Model Providers |
| Mixtral 8x7B | Mistral logo | Open | 33k | 41 | 1114 | 0.63 | 0.30 | 0.31 | 0.38 | – | – | $0.50 | $0.45 | $0.50 | 110.1 | 56.9 | 89.9 | 149.1 | 550.3 | 0.36 | 0.21 | 0.29 | 0.48 | 0.69 | Model Providers |
| Codestral-Mamba | Mistral logo | Open | 256k | 33 | – | 0.25 | 0.03 | 0.24 | 0.79 | – | – | $0.25 | $0.25 | $0.25 | 95.8 | 89.5 | 92.8 | 96.9 | 98.3 | 0.44 | 0.38 | 0.41 | 0.48 | 0.68 | Model Providers |
| Command-R+ | Cohere logo | Open | 128k | 55 | 1215 | 0.75 | 0.34 | 0.40 | 0.71 | – | – | $5.19 | $2.75 | $12.50 | 50.7 | 45.4 | 47.5 | 73.9 | 79.7 | 0.47 | 0.23 | 0.27 | 0.51 | 0.64 | Model Providers |
| Command-R+ (Apr '24) | Cohere logo | Open | 128k | 45 | 1190 | 0.68 | 0.24 | 0.27 | 0.62 | – | – | $6.00 | $3.00 | $15.00 | 49.3 | 45.6 | 47.4 | 64.9 | 77.2 | 0.51 | 0.25 | 0.34 | 0.58 | 0.64 | Model Providers |
| Command-R (Mar '24) | Cohere logo | Open | 128k | 36 | 1149 | 0.59 | 0.26 | 0.16 | 0.44 | – | – | $0.75 | $0.50 | $1.50 | 108.2 | 75.9 | 80.3 | 167.6 | 178.4 | 0.36 | 0.15 | 0.25 | 0.44 | 0.50 | Model Providers |
| Aya Expanse 8B | Cohere logo | Open | 8k | – | – | – | – | – | – | – | – | $0.75 | $0.50 | $1.50 | 165.6 | 157.5 | 162.0 | 169.3 | 173.7 | 0.16 | 0.12 | 0.14 | 0.21 | 0.32 | Model Providers |
| Command-R | Cohere logo | Open | 128k | – | 1179 | 0.67 | 0.27 | – | 0.70 | – | – | $0.51 | $0.33 | $1.05 | 111.8 | 102.7 | 107.6 | 117.7 | 124.9 | 0.32 | 0.15 | 0.21 | 0.36 | 0.39 | Model Providers |
| Aya Expanse 32B | Cohere logo | Open | 128k | – | 1207 | 0.67 | – | – | – | – | – | $0.75 | $0.50 | $1.50 | 120.3 | 114.9 | 118.4 | 123.3 | 127.5 | 0.18 | 0.15 | 0.16 | 0.25 | 0.32 | Model Providers |
| Sonar 3.1 Small | Perplexity logo | Proprietary | 127k | – | – | – | – | – | – | – | – | $0.20 | $0.20 | $0.20 | 203.8 | 182.9 | 201.0 | 205.5 | 206.9 | 0.35 | 0.29 | 0.30 | 0.37 | 0.47 | Model Providers |
| Sonar 3.1 Large | Perplexity logo | Proprietary | 127k | – | – | – | – | – | – | – | – | $1.00 | $1.00 | $1.00 | 57.8 | 45.7 | 54.0 | 60.8 | 64.6 | 0.31 | 0.29 | 0.30 | 0.36 | 0.46 | Model Providers |
| Grok Beta | xAI logo | Proprietary | 128k | 72 | 1289 | 0.85 | 0.43 | 0.73 | 0.87 | – | – | $7.50 | $5.00 | $15.00 | 66.5 | 56.9 | 64.4 | 67.6 | 68.8 | 0.42 | 0.34 | 0.38 | 0.47 | 0.56 | Model Providers |
| Nova Pro | Amazon logo | Proprietary | 300k | 75 | – | 0.84 | 0.48 | 0.79 | 0.88 | – | – | $1.40 | $0.80 | $3.20 | 91.3 | 77.6 | 82.4 | 96.4 | 102.5 | 0.38 | 0.35 | 0.37 | 0.39 | 0.42 | Model Providers |
| Nova Lite | Amazon logo | Proprietary | 300k | 70 | – | 0.79 | 0.43 | 0.75 | 0.84 | – | – | $0.10 | $0.06 | $0.24 | 148.0 | 126.4 | 134.8 | 156.7 | 165.5 | 0.33 | 0.30 | 0.32 | 0.35 | 0.38 | Model Providers |
| Nova Micro | Amazon logo | Proprietary | 130k | 66 | – | 0.76 | 0.38 | 0.69 | 0.80 | – | – | $0.06 | $0.04 | $0.14 | 195.8 | 170.9 | 186.0 | 208.3 | 219.5 | 0.33 | 0.30 | 0.32 | 0.35 | 0.39 | Model Providers |
| Phi-4 | Microsoft Azure logo | Open | 16k | 77 | – | 0.85 | 0.57 | 0.81 | 0.87 | – | – | $0.09 | $0.07 | $0.14 | 85.1 | 76.2 | 82.0 | 85.4 | 85.6 | 0.21 | 0.16 | 0.18 | 0.23 | 0.25 | Model Providers |
| Phi-3 Mini | Microsoft Azure logo | Open | 4k | – | 1037 | – | – | – | – | – | – | $0.00 | $0.00 | $0.00 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| Phi-3 Medium 14B | Microsoft Azure logo | Open | 128k | – | 1123 | – | – | – | – | – | – | $0.30 | $0.17 | $0.68 | 50.8 | 17.6 | 44.9 | 52.7 | 54.2 | 0.43 | 0.38 | 0.42 | 0.45 | 0.49 | Model Providers |
| Solar Mini | Upstage logo | Open | 4k | 47 | 1062 | 0.66 | 0.28 | 0.33 | 0.59 | – | – | $0.15 | $0.15 | $0.15 | 89.1 | 84.3 | 87.7 | 90.5 | 92.5 | 1.12 | 1.07 | 1.12 | 1.14 | 1.42 | Model Providers |
| DBRX | Databricks logo | Open | 33k | 46 | 1103 | 0.70 | 0.31 | 0.28 | 0.54 | – | – | $1.16 | $0.97 | $1.73 | 74.2 | 50.2 | 68.0 | 82.9 | 83.1 | 0.44 | 0.27 | 0.32 | 0.51 | 0.71 | Model Providers |
| Llama 3.1 Nemotron 70B | NVIDIA logo | Open | 128k | 72 | 1269 | 0.86 | 0.48 | 0.73 | 0.81 | – | – | $0.27 | $0.23 | $0.40 | 48.3 | 27.4 | 44.3 | 69.8 | 71.0 | 0.57 | 0.23 | 0.32 | 0.64 | 0.79 | Model Providers |
| Reka Flash | Reka AI logo | Proprietary | 128k | 59 | – | 0.73 | 0.34 | 0.53 | 0.74 | – | – | $0.35 | $0.20 | $0.80 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| Reka Core | Reka AI logo | Proprietary | 128k | 58 | 1199 | 0.76 | 0.28 | 0.56 | 0.73 | – | – | $2.00 | $2.00 | $2.00 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| Reka Flash (Feb '24) | Reka AI logo | Proprietary | 128k | 46 | 1165 | 0.65 | 0.27 | 0.33 | 0.61 | – | – | $0.35 | $0.20 | $0.80 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| Reka Edge | Reka AI logo | Proprietary | 128k | 31 | – | 0.44 | 0.19 | 0.22 | 0.41 | – | – | $0.10 | $0.10 | $0.10 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| Jamba 1.5 Large | AI21 Labs logo | Open | 256k | 64 | 1221 | 0.80 | 0.41 | 0.61 | 0.74 | – | – | $3.50 | $2.00 | $8.00 | 50.8 | 43.9 | 49.9 | 51.8 | 58.4 | 0.72 | 0.49 | 0.68 | 0.74 | 0.81 | Model Providers |
| Jamba 1.5 Mini | AI21 Labs logo | Open | 256k | – | 1176 | 0.63 | 0.26 | 0.61 | – | – | – | $0.25 | $0.20 | $0.40 | 83.5 | 78.0 | 82.0 | 164.1 | 190.3 | 0.48 | 0.29 | 0.44 | 0.51 | 0.55 | Model Providers |
| DeepSeek V3 | DeepSeek logo | Open | 128k | 80 | – | 0.87 | 0.56 | 0.85 | 0.92 | – | – | $0.90 | $0.90 | $1.10 | 21.1 | 7.9 | 10.8 | 48.3 | 73.1 | 0.95 | 0.52 | 0.67 | 1.24 | 10.32 | Model Providers |
| DeepSeek-V2.5 (Dec '24) | DeepSeek logo | Open | 128k | 72 | – | 0.81 | 0.42 | 0.76 | 0.88 | – | – | $0.17 | $0.14 | $0.28 | 64.8 | 52.7 | 57.3 | 70.9 | 81.1 | 1.16 | 0.90 | 1.02 | 1.37 | 1.87 | Model Providers |
| DeepSeek-Coder-V2 | DeepSeek logo | Open | 128k | 71 | 1178 | 0.80 | 0.42 | 0.74 | 0.87 | – | – | $0.17 | $0.14 | $0.28 | 64.4 | 51.8 | 57.3 | 71.4 | 81.1 | 1.12 | 0.84 | 0.99 | 1.27 | 1.71 | Model Providers |
| DeepSeek-V2.5 | DeepSeek logo | Open | 128k | – | 1258 | 0.81 | 0.42 | – | 0.87 | – | – | $1.09 | $1.07 | $1.14 | 7.6 | 6.9 | 7.2 | 8.0 | 8.2 | 0.77 | 0.60 | 0.70 | 0.88 | 17.15 | Model Providers |
| DeepSeek-V2 | DeepSeek logo | Open | 128k | – | 1220 | 0.80 | 0.42 | – | 0.87 | – | – | $0.17 | $0.14 | $0.28 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| Arctic | Snowflake logo | Open | 4k | 51 | 1090 | 0.63 | 0.26 | – | 0.75 | – | – | $0.00 | $0.00 | $0.00 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| Qwen2.5 72B | Alibaba logo | Open | 131k | 77 | 1259 | 0.86 | 0.50 | 0.84 | 0.89 | – | – | $0.40 | $0.40 | $0.75 | 65.6 | 18.5 | 39.6 | 80.8 | 242.6 | 0.54 | 0.25 | 0.38 | 0.68 | 2.57 | Model Providers |
| Qwen2.5 Coder 32B | Alibaba logo | Open | 131k | 72 | 1220 | 0.79 | 0.41 | 0.76 | 0.91 | – | – | $0.80 | $0.80 | $0.80 | 84.1 | 34.7 | 45.4 | 102.8 | 344.0 | 0.37 | 0.21 | 0.28 | 0.51 | 1.25 | Model Providers |
| Qwen2 72B | Alibaba logo | Open | 131k | 72 | 1187 | 0.83 | 0.40 | 0.77 | 0.86 | – | – | $0.63 | $0.63 | $0.65 | 45.7 | 32.8 | 37.7 | 64.2 | 67.0 | 0.30 | 0.23 | 0.27 | 0.36 | 0.62 | Model Providers |
| QwQ 32B-Preview | Alibaba logo | Open | 33k | 46 | – | 0.07 | 0.01 | 0.91 | 0.85 | – | – | $0.26 | $0.20 | $0.60 | 66.3 | 35.0 | 52.1 | 105.2 | 329.9 | 0.40 | 0.21 | 0.32 | 0.58 | 2.05 | Model Providers |
| Yi-Large | 01.AI logo | Proprietary | 32k | 61 | 1213 | 0.78 | 0.33 | 0.56 | 0.77 | – | – | $3.00 | $3.00 | $3.00 | 67.4 | 63.1 | 64.5 | 78.9 | 81.8 | 0.47 | 0.35 | 0.40 | 0.78 | 1.64 | Model Providers |
| GPT-4 Turbo | OpenAI logo | Proprietary | 128k | 75 | 1256 | 0.87 | 0.50 | 0.74 | 0.92 | – | – | $15.00 | $10.00 | $30.00 | 43.8 | 22.7 | 35.3 | 52.8 | 58.7 | 1.19 | 0.51 | 0.63 | 1.44 | 2.02 | Model Providers |
| GPT-4 | OpenAI logo | Proprietary | 8k | – | 1186 | – | – | – | – | – | – | $37.50 | $30.00 | $60.00 | 30.5 | 14.2 | 20.2 | 37.9 | 44.9 | 0.73 | 0.50 | 0.57 | 0.91 | 1.30 | Model Providers |
| Llama 3 70B | Meta logo | Open | 8k | 48 | 1206 | 0.79 | 0.39 | 0.53 | 0.19 | – | – | $0.88 | $0.80 | $0.88 | 48.2 | 19.3 | 32.9 | 130.8 | 349.0 | 0.40 | 0.23 | 0.30 | 0.55 | 1.36 | Model Providers |
| Llama 3 8B | Meta logo | Open | 8k | 45 | 1152 | 0.64 | 0.30 | 0.32 | 0.53 | – | – | $0.10 | $0.07 | $0.20 | 109.4 | 63.1 | 74.0 | 202.4 | 1203.6 | 0.35 | 0.19 | 0.30 | 0.41 | 0.73 | Model Providers |
| Llama 2 Chat 70B | Meta logo | Open | 4k | – | 1093 | – | – | – | – | – | – | $1.85 | $1.75 | $2.17 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| Llama 2 Chat 13B | Meta logo | Open | 4k | – | 1063 | – | – | – | – | – | – | $0.00 | $0.00 | $0.00 | – | – | – | – | – | – | – | – | – | – | Model Providers |
| Llama 2 Chat 7B | Meta logo | Open | 4k | – | 1037 | – | – | – | – | – | – | $0.33 | $0.29 | $0.46 | 123.8 | 119.1 | 122.3 | 126.0 | 130.1 | 0.37 | 0.34 | 0.35 | 0.40 | 0.55 | Model Providers |
| Gemini 1.0 Pro | Google logo | Proprietary | 33k | – | 1111 | – | – | – | – | – | – | $0.75 | $0.50 | $1.50 | 103.1 | 97.1 | 98.7 | 105.2 | 107.5 | 1.28 | 1.20 | 1.24 | 1.31 | 1.39 | Model Providers |
| Claude 3 Sonnet | Anthropic logo | Proprietary | 200k | 57 | 1201 | 0.77 | 0.37 | 0.41 | 0.71 | – | – | $6.00 | $3.00 | $15.00 | 67.9 | 35.2 | 58.5 | 76.6 | 89.7 | 0.75 | 0.61 | 0.65 | 0.86 | 1.46 | Model Providers |
| Claude 2.1 | Anthropic logo | Proprietary | 200k | – | 1118 | – | – | – | – | – | – | $12.00 | $8.00 | $24.00 | 20.3 | 13.0 | 13.5 | 29.4 | 31.0 | 1.41 | 0.79 | 0.82 | 1.78 | 2.01 | Model Providers |
| Claude 2.0 | Anthropic logo | Proprietary | 100k | – | 1132 | – | – | – | – | – | – | $12.00 | $8.00 | $24.00 | 29.9 | 28.7 | 29.4 | 30.5 | 32.6 | 0.81 | 0.78 | 0.80 | 0.83 | 0.95 | Model Providers |
| Mistral Small (Feb '24) | Mistral logo | Proprietary | 33k | 59 | – | 0.69 | 0.31 | 0.56 | 0.79 | – | – | $1.50 | $1.00 | $3.00 | 53.7 | 48.8 | 52.1 | 61.6 | 73.2 | 0.38 | 0.25 | 0.33 | 0.41 | 0.62 | Model Providers |
| Mistral Large (Feb '24) | Mistral logo | Proprietary | 33k | 56 | 1157 | 0.69 | 0.36 | 0.49 | 0.70 | – | – | $6.00 | $4.00 | $12.00 | 38.8 | 29.8 | 36.8 | 42.5 | 45.4 | 0.43 | 0.34 | 0.40 | 0.52 | 0.98 | Model Providers |
| Mistral 7B | Mistral logo | Open | 8k | 28 | 1008 | 0.33 | 0.19 | 0.16 | 0.42 | – | – | $0.12 | $0.11 | $0.14 | 101.4 | 76.1 | 92.8 | 127.4 | 156.1 | 0.30 | 0.15 | 0.22 | 0.35 | 0.96 | Model Providers |
| Mistral Medium | Mistral logo | Proprietary | 33k | – | 1148 | – | – | – | – | – | – | $4.09 | $2.75 | $8.10 | 44.5 | 40.3 | 42.7 | 45.3 | 46.8 | 0.38 | 0.31 | 0.33 | 0.45 | 17.22 | Model Providers |
| Codestral | Mistral logo | Open | 33k | – | – | 0.23 | – | 0.80 | – | – | – | $0.30 | $0.20 | $0.60 | 84.8 | 78.1 | 82.1 | 86.5 | 88.8 | 0.28 | 0.24 | 0.26 | 0.31 | 0.49 | Model Providers |
| OpenChat 3.5 | OpenChat logo | Open | 8k | 44 | 1076 | 0.56 | 0.22 | 0.31 | 0.68 | – | – | $0.06 | $0.06 | $0.06 | 73.3 | 66.3 | 69.3 | 76.3 | 80.3 | 0.30 | 0.24 | 0.27 | 0.32 | 0.37 | Model Providers |
| Jamba Instruct | AI21 Labs logo | Proprietary | 256k | – | – | 0.58 | 0.25 | – | – | – | – | $0.55 | $0.50 | $0.70 | 77.1 | 70.4 | 74.3 | 169.6 | 193.7 | 0.52 | 0.29 | 0.45 | 0.54 | 0.58 | Model Providers |
Terms
- Artificial Analysis Quality Index: Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details.
- Context window': Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).
- Output Speed': Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).
- Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.
- Price': Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).
- Output Price': Price per token generated by the model (received from the API), represented as USD per million Tokens.
- Input Price': Price per token included in the request/message sent to the API, represented as USD per million Tokens.
- Time period': Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.