LLM Comparisons: Difference between revisions

From AI Wiki
No edit summary
Line 7: Line 7:
! Context Window
! Context Window
! Quality Index
! Quality Index
! Normalized avg
! Blended
! Blended
! USD/1M Tokens
! Median Tokens/s
! Median Tokens/s
! Median First Chunk (s)
! Median First Chunk (s)
! Further Analysis
|-
|-
| '''[[o1-preview]]'''
| '''[[o1-preview]]'''
| OpenAI logo
| [[OpenAI]]
| 128k
| 128k
| 86
| 86
| –
| $27.56
| $27.56
| –
| 144.5
| 144.5
| 21.12
| 21.12
| Model Providers
|-
|-
| '''[[o1-mini]]'''
| '''[[o1-mini]]'''
| OpenAI logo
| [[OpenAI]]
| 128k
| 128k
| 84
| 84
| –
| $5.25
| $5.25
| –
| 213.2
| 213.2
| 11.64
| 11.64
| Model Providers
|-
|-
| '''[[GPT-4o (Aug '24)]]'''
| '''[[GPT-4o (Aug '24)]]'''
| OpenAI logo
| [[OpenAI]]
| 128k
| 128k
| 78
| 78
| –
| $4.38
| $4.38
| –
| 85.6
| 85.6
| 0.65
| 0.65
| Model Providers
|-
|-
| '''[[GPT-4o (May '24)]]'''
| '''[[GPT-4o (May '24)]]'''
| OpenAI logo
| [[OpenAI]]
| 128k
| 128k
| 78
| 78
| –
| $7.50
| $7.50
| –
| 106.6
| 106.6
| 0.65
| 0.65
| Model Providers
|-
|-
| '''[[GPT-4o mini]]'''
| '''[[GPT-4o mini]]'''
| OpenAI logo
| [[OpenAI]]
| 128k
| 128k
| 73
| 73
| –
| $0.26
| $0.26
| –
| 121.8
| 121.8
| 0.65
| 0.65
| Model Providers
|-
|-
| '''[[GPT-4o (Nov '24)]]'''
| '''[[GPT-4o (Nov '24)]]'''
| OpenAI logo
| [[OpenAI]]
| 128k
| 128k
| 73
| 73
| –
| $4.38
| $4.38
| –
| 115.7
| 115.7
| 0.38
| 0.38
| Model Providers
|-
|-
| '''[[GPT-4o mini Realtime (Dec '24)]]'''
| '''[[GPT-4o mini Realtime (Dec '24)]]'''
| OpenAI logo
| [[OpenAI]]
| 128k
| 128k
| –
| –
| –
| $0.00
| $0.00
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[GPT-4o Realtime (Dec '24)]]'''
| '''[[GPT-4o Realtime (Dec '24)]]'''
| OpenAI logo
| [[OpenAI]]
| 128k
| 128k
| –
| –
| –
| $0.00
| $0.00
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[Llama 3.3 70B]]'''
| '''[[Llama 3.3 70B]]'''
| Meta logo
| [[Meta]]
| 128k
| 128k
| 74
| 74
| –
| $0.69
| $0.69
| –
| 71.3
| 71.3
| 0.50
| 0.50
| Model Providers
|-
|-
| '''[[Llama 3.1 405B]]'''
| '''[[Llama 3.1 405B]]'''
| Meta logo
| [[Meta]]
| 128k
| 128k
| 74
| 74
| –
| $3.50
| $3.50
| –
| 30.2
| 30.2
| 0.72
| 0.72
| Model Providers
|-
|-
| '''[[Llama 3.1 70B]]'''
| '''[[Llama 3.1 70B]]'''
| Meta logo
| [[Meta]]
| 128k
| 128k
| 68
| 68
| –
| $0.72
| $0.72
| –
| 72.7
| 72.7
| 0.44
| 0.44
| Model Providers
|-
|-
| '''[[Llama 3.2 90B (Vision)]]'''
| '''[[Llama 3.2 90B (Vision)]]'''
| Meta logo
| [[Meta]]
| 128k
| 128k
| 68
| 68
| –
| $0.81
| $0.81
| –
| 48.5
| 48.5
| 0.34
| 0.34
| Model Providers
|-
|-
| '''[[Llama 3.2 11B (Vision)]]'''
| '''[[Llama 3.2 11B (Vision)]]'''
| Meta logo
| [[Meta]]
| 128k
| 128k
| 54
| 54
| –
| $0.18
| $0.18
| –
| 131.0
| 131.0
| 0.29
| 0.29
| Model Providers
|-
|-
| '''[[Llama 3.1 8B]]'''
| '''[[Llama 3.1 8B]]'''
| Meta logo
| [[Meta]]
| 128k
| 128k
| 54
| 54
| –
| $0.10
| $0.10
| –
| 184.7
| 184.7
| 0.34
| 0.34
| Model Providers
|-
|-
| '''[[Llama 3.2 3B]]'''
| '''[[Llama 3.2 3B]]'''
| Meta logo
| [[Meta]]
| 128k
| 128k
| 49
| 49
| –
| $0.06
| $0.06
| –
| 202.1
| 202.1
| 0.38
| 0.38
| Model Providers
|-
|-
| '''[[Llama 3.2 1B]]'''
| '''[[Llama 3.2 1B]]'''
| Meta logo
| [[Meta]]
| 128k
| 128k
| 26
| 26
| –
| $0.04
| $0.04
| –
| 468.9
| 468.9
| 0.38
| 0.38
| Model Providers
|-
|-
| '''[[Gemini 2.0 Flash (exp)]]'''
| '''[[Gemini 2.0 Flash (exp)]]'''
| Google logo
| [[Google]]
| 1m
| 1m
| 82
| 82
| –
| $0.00
| $0.00
| –
| 169.0
| 169.0
| 0.48
| 0.48
| Model Providers
|-
|-
| '''[[Gemini 1.5 Pro (Sep)]]'''
| '''[[Gemini 1.5 Pro (Sep)]]'''
| Google logo
| [[Google]]
| 2m
| 2m
| 80
| 80
| –
| $2.19
| $2.19
| –
| 60.9
| 60.9
| 0.74
| 0.74
| Model Providers
|-
|-
| '''[[Gemini 1.5 Flash (Sep)]]'''
| '''[[Gemini 1.5 Flash (Sep)]]'''
| Google logo
| [[Google]]
| 1m
| 1m
| 72
| 72
| –
| $0.13
| $0.13
| –
| 188.6
| 188.6
| 0.25
| 0.25
| Model Providers
|-
|-
| '''[[Gemma 2 27B]]'''
| '''[[Gemma 2 27B]]'''
| Google logo
| [[Google]]
| 8k
| 8k
| 61
| 61
| –
| $0.26
| $0.26
| –
| 58.9
| 58.9
| 0.51
| 0.51
| Model Providers
|-
|-
| '''[[Gemma 2 9B]]'''
| '''[[Gemma 2 9B]]'''
| Google logo
| [[Google]]
| 8k
| 8k
| 55
| 55
| –
| $0.12
| $0.12
| –
| 168.9
| 168.9
| 0.37
| 0.37
| Model Providers
|-
|-
| '''[[Gemini 1.5 Flash (May)]]'''
| '''[[Gemini 1.5 Flash (May)]]'''
| Google logo
| [[Google]]
| 1m
| 1m
| –
| –
| –
| $0.13
| $0.13
| –
| 310.4
| 310.4
| 0.29
| 0.29
| Model Providers
|-
|-
| '''[[Gemini Experimental (Nov)]]'''
| '''[[Gemini Experimental (Nov)]]'''
| Google logo
| [[Google]]
| 2m
| 2m
| –
| –
| –
| $0.00
| $0.00
| –
| 53.9
| 53.9
| 1.11
| 1.11
| Model Providers
|-
|-
| '''[[Gemini 1.5 Pro (May)]]'''
| '''[[Gemini 1.5 Pro (May)]]'''
| Google logo
| [[Google]]
| 2m
| 2m
| –
| –
| –
| $2.19
| $2.19
| –
| 66.9
| 66.9
| 0.50
| 0.50
| Model Providers
|-
|-
| '''[[Gemini 1.5 Flash-8B]]'''
| '''[[Gemini 1.5 Flash-8B]]'''
| Google logo
| [[Google]]
| 1m
| 1m
| –
| –
| –
| $0.07
| $0.07
| –
| 278.8
| 278.8
| 0.39
| 0.39
| Model Providers
|-
|-
| '''[[Claude 3.5 Sonnet (Oct)]]'''
| '''[[Claude 3.5 Sonnet (Oct)]]'''
| Anthropic logo
| [[Anthropic]]
| 200k
| 200k
| 80
| 80
| –
| $6.00
| $6.00
| –
| 71.9
| 71.9
| 0.98
| 0.98
| Model Providers
|-
|-
| '''[[Claude 3.5 Sonnet (June)]]'''
| '''[[Claude 3.5 Sonnet (June)]]'''
| Anthropic logo
| [[Anthropic]]
| 200k
| 200k
| 76
| 76
| –
| $6.00
| $6.00
| –
| 61.4
| 61.4
| 0.87
| 0.87
| Model Providers
|-
|-
| '''[[Claude 3 Opus]]'''
| '''[[Claude 3 Opus]]'''
| Anthropic logo
| [[Anthropic]]
| 200k
| 200k
| 70
| 70
| –
| $30.00
| $30.00
| –
| 25.9
| 25.9
| 2.00
| 2.00
| Model Providers
|-
|-
| '''[[Claude 3.5 Haiku]]'''
| '''[[Claude 3.5 Haiku]]'''
| Anthropic logo
| [[Anthropic]]
| 200k
| 200k
| 68
| 68
| –
| $1.60
| $1.60
| –
| 65.1
| 65.1
| 0.71
| 0.71
| Model Providers
|-
|-
| '''[[Claude 3 Haiku]]'''
| '''[[Claude 3 Haiku]]'''
| Anthropic logo
| [[Anthropic]]
| 200k
| 200k
| 55
| 55
| –
| $0.50
| $0.50
| –
| 122.7
| 122.7
| 0.72
| 0.72
| Model Providers
|-
|-
| '''[[Pixtral Large]]'''
| '''[[Pixtral Large]]'''
| Mistral logo
| [[Mistral]]
| 128k
| 128k
| 74
| 74
| –
| $3.00
| $3.00
| –
| 36.7
| 36.7
| 0.39
| 0.39
| Model Providers
|-
|-
| '''[[Mistral Large 2 (Jul '24)]]'''
| '''[[Mistral Large 2 (Jul '24)]]'''
| Mistral logo
| [[Mistral]]
| 128k
| 128k
| 74
| 74
| –
| $3.00
| $3.00
| –
| 31.2
| 31.2
| 0.50
| 0.50
| Model Providers
|-
|-
| '''[[Mistral Large 2 (Nov '24)]]'''
| '''[[Mistral Large 2 (Nov '24)]]'''
| Mistral logo
| [[Mistral]]
| 128k
| 128k
| 74
| 74
| –
| $3.00
| $3.00
| –
| 37.4
| 37.4
| 0.52
| 0.52
| Model Providers
|-
|-
| '''[[Mistral Small (Sep '24)]]'''
| '''[[Mistral Small (Sep '24)]]'''
| Mistral logo
| [[Mistral]]
| 33k
| 33k
| 61
| 61
| –
| $0.30
| $0.30
| –
| 61.5
| 61.5
| 0.32
| 0.32
| Model Providers
|-
|-
| '''[[Mixtral 8x22B]]'''
| '''[[Mixtral 8x22B]]'''
| Mistral logo
| [[Mistral]]
| 65k
| 65k
| 61
| 61
| –
| $1.20
| $1.20
| –
| 85.5
| 85.5
| 0.57
| 0.57
| Model Providers
|-
|-
| '''[[Pixtral 12B]]'''
| '''[[Pixtral 12B]]'''
| Mistral logo
| [[Mistral]]
| 128k
| 128k
| 56
| 56
| –
| $0.13
| $0.13
| –
| 70.5
| 70.5
| 0.37
| 0.37
| Model Providers
|-
|-
| '''[[Ministral 8B]]'''
| '''[[Ministral 8B]]'''
| Mistral logo
| [[Mistral]]
| 128k
| 128k
| 56
| 56
| –
| $0.10
| $0.10
| –
| 135.8
| 135.8
| 0.29
| 0.29
| Model Providers
|-
|-
| '''[[Mistral NeMo]]'''
| '''[[Mistral NeMo]]'''
| Mistral logo
| [[Mistral]]
| 128k
| 128k
| 54
| 54
| –
| $0.09
| $0.09
| –
| 123.7
| 123.7
| 0.48
| 0.48
| Model Providers
|-
|-
| '''[[Ministral 3B]]'''
| '''[[Ministral 3B]]'''
| Mistral logo
| [[Mistral]]
| 128k
| 128k
| 53
| 53
| –
| $0.04
| $0.04
| –
| 168.1
| 168.1
| 0.29
| 0.29
| Model Providers
|-
|-
| '''[[Mixtral 8x7B]]'''
| '''[[Mixtral 8x7B]]'''
| Mistral logo
| [[Mistral]]
| 33k
| 33k
| 41
| 41
| –
| $0.50
| $0.50
| –
| 110.1
| 110.1
| 0.36
| 0.36
| Model Providers
|-
|-
| '''[[Codestral-Mamba]]'''
| '''[[Codestral-Mamba]]'''
| Mistral logo
| [[Mistral]]
| 256k
| 256k
| 33
| 33
| –
| $0.25
| $0.25
| –
| 95.8
| 95.8
| 0.44
| 0.44
| Model Providers
|-
|-
| '''[[Command-R+]]'''
| '''[[Command-R+]]'''
| Cohere logo
| [[Cohere]]
| 128k
| 128k
| 55
| 55
| –
| $5.19
| $5.19
| –
| 50.7
| 50.7
| 0.47
| 0.47
| Model Providers
|-
|-
| '''[[Command-R+ (Apr '24)]]'''
| '''[[Command-R+ (Apr '24)]]'''
| Cohere logo
| [[Cohere]]
| 128k
| 128k
| 45
| 45
| –
| $6.00
| $6.00
| –
| 49.3
| 49.3
| 0.51
| 0.51
| Model Providers
|-
|-
| '''[[Command-R (Mar '24)]]'''
| '''[[Command-R (Mar '24)]]'''
| Cohere logo
| [[Cohere]]
| 128k
| 128k
| 36
| 36
| –
| $0.75
| $0.75
| –
| 108.3
| 108.3
| 0.36
| 0.36
| Model Providers
|-
|-
| '''[[Aya Expanse 8B]]'''
| '''[[Aya Expanse 8B]]'''
| Cohere logo
| [[Cohere]]
| 8k
| 8k
| –
| –
| –
| $0.75
| $0.75
| –
| 165.6
| 165.6
| 0.16
| 0.16
| Model Providers
|-
|-
| '''[[Command-R]]'''
| '''[[Command-R]]'''
| Cohere logo
| [[Cohere]]
| 128k
| 128k
| –
| –
| –
| $0.51
| $0.51
| –
| 111.7
| 111.7
| 0.32
| 0.32
| Model Providers
|-
|-
| '''[[Aya Expanse 32B]]'''
| '''[[Aya Expanse 32B]]'''
| Cohere logo
| [[Cohere]]
| 128k
| 128k
| –
| –
| –
| $0.75
| $0.75
| –
| 120.3
| 120.3
| 0.18
| 0.18
| Model Providers
|-
|-
| '''[[Sonar 3.1 Small]]'''
| '''[[Sonar 3.1 Small]]'''
| Perplexity logo
| [[Perplexity]]
| 127k
| 127k
| –
| –
| –
| $0.20
| $0.20
| –
| 203.8
| 203.8
| 0.35
| 0.35
| Model Providers
|-
|-
| '''[[Sonar 3.1 Large]]'''
| '''[[Sonar 3.1 Large]]'''
| Perplexity logo
| [[Perplexity]]
| 127k
| 127k
| –
| –
| –
| $1.00
| $1.00
| –
| 57.8
| 57.8
| 0.31
| 0.31
| Model Providers
|-
|-
| '''[[Grok Beta]]'''
| '''[[Grok Beta]]'''
| xAI logo
| [[xAI]]
| 128k
| 128k
| 72
| 72
| –
| $7.50
| $7.50
| –
| 66.5
| 66.5
| 0.42
| 0.42
| Model Providers
|-
|-
| '''[[Nova Pro]]'''
| '''[[Nova Pro]]'''
| Amazon logo
| [[Amazon]]
| 300k
| 300k
| 75
| 75
| –
| $1.40
| $1.40
| –
| 91.1
| 91.1
| 0.38
| 0.38
| Model Providers
|-
|-
| '''[[Nova Lite]]'''
| '''[[Nova Lite]]'''
| Amazon logo
| [[Amazon]]
| 300k
| 300k
| 70
| 70
| –
| $0.10
| $0.10
| –
| 147.7
| 147.7
| 0.34
| 0.34
| Model Providers
|-
|-
| '''[[Nova Micro]]'''
| '''[[Nova Micro]]'''
| Amazon logo
| [[Amazon]]
| 130k
| 130k
| 66
| 66
| –
| $0.06
| $0.06
| –
| 195.3
| 195.3
| 0.33
| 0.33
| Model Providers
|-
|-
| '''[[Phi-4]]'''
| '''[[Phi-4]]'''
| Microsoft Azure logo
| [[Microsoft Azure]]
| 16k
| 16k
| 77
| 77
| –
| $0.09
| $0.09
| –
| 85.1
| 85.1
| 0.21
| 0.21
| Model Providers
|-
|-
| '''[[Phi-3 Mini]]'''
| '''[[Phi-3 Mini]]'''
| Microsoft Azure logo
| [[Microsoft Azure]]
| 4k
| 4k
| –
| –
| –
| $0.00
| $0.00
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[Phi-3 Medium 14B]]'''
| '''[[Phi-3 Medium 14B]]'''
| Microsoft Azure logo
| [[Microsoft Azure]]
| 128k
| 128k
| –
| –
| –
| $0.30
| $0.30
| –
| 50.7
| 50.7
| 0.43
| 0.43
| Model Providers
|-
|-
| '''[[Solar Mini]]'''
| '''[[Solar Mini]]'''
| Upstage logo
| [[Upstage]]
| 4k
| 4k
| 47
| 47
| –
| $0.15
| $0.15
| –
| 89.3
| 89.3
| 1.12
| 1.12
| Model Providers
|-
|-
| '''[[DBRX]]'''
| '''[[DBRX]]'''
| Databricks logo
| [[Databricks]]
| 33k
| 33k
| 46
| 46
| –
| $1.16
| $1.16
| –
| 74.2
| 74.2
| 0.43
| 0.43
| Model Providers
|-
|-
| '''[[Llama 3.1 Nemotron 70B]]'''
| '''[[Llama 3.1 Nemotron 70B]]'''
| NVIDIA logo
| [[NVIDIA]]
| 128k
| 128k
| 72
| 72
| –
| $0.27
| $0.27
| –
| 48.3
| 48.3
| 0.57
| 0.57
| Model Providers
|-
|-
| '''[[Reka Flash]]'''
| '''[[Reka Flash]]'''
| Reka AI logo
| [[Reka AI]]
| 128k
| 128k
| 59
| 59
| –
| $0.35
| $0.35
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[Reka Core]]'''
| '''[[Reka Core]]'''
| Reka AI logo
| [[Reka AI]]
| 128k
| 128k
| 58
| 58
| –
| $2.00
| $2.00
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[Reka Flash (Feb '24)]]'''
| '''[[Reka Flash (Feb '24)]]'''
| Reka AI logo
| [[Reka AI]]
| 128k
| 128k
| 46
| 46
| –
| $0.35
| $0.35
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[Reka Edge]]'''
| '''[[Reka Edge]]'''
| Reka AI logo
| [[Reka AI]]
| 128k
| 128k
| 31
| 31
| –
| $0.10
| $0.10
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[Jamba 1.5 Large]]'''
| '''[[Jamba 1.5 Large]]'''
| AI21 Labs logo
| [[AI21 Labs]]
| 256k
| 256k
| 64
| 64
| –
| $3.50
| $3.50
| –
| 50.8
| 50.8
| 0.71
| 0.71
| Model Providers
|-
|-
| '''[[Jamba 1.5 Mini]]'''
| '''[[Jamba 1.5 Mini]]'''
| AI21 Labs logo
| [[AI21 Labs]]
| 256k
| 256k
| –
| –
| –
| $0.25
| $0.25
| –
| 83.5
| 83.5
| 0.48
| 0.48
| Model Providers
|-
|-
| '''[[DeepSeek V3]]'''
| '''[[DeepSeek V3]]'''
| DeepSeek logo
| [[DeepSeek]]
| 128k
| 128k
| 80
| 80
| –
| $0.90
| $0.90
| –
| 21.1
| 21.1
| 0.95
| 0.95
| Model Providers
|-
|-
| '''[[DeepSeek-V2.5 (Dec '24)]]'''
| '''[[DeepSeek-V2.5 (Dec '24)]]'''
| DeepSeek logo
| [[DeepSeek]]
| 128k
| 128k
| 72
| 72
| –
| $0.17
| $0.17
| –
| 63.8
| 63.8
| 1.15
| 1.15
| Model Providers
|-
|-
| '''[[DeepSeek-Coder-V2]]'''
| '''[[DeepSeek-Coder-V2]]'''
| DeepSeek logo
| [[DeepSeek]]
| 128k
| 128k
| 71
| 71
| –
| $0.17
| $0.17
| –
| 63.8
| 63.8
| 1.12
| 1.12
| Model Providers
|-
|-
| '''[[DeepSeek-V2.5]]'''
| '''[[DeepSeek-V2.5]]'''
| DeepSeek logo
| [[DeepSeek]]
| 128k
| 128k
| –
| –
| –
| $1.09
| $1.09
| –
| 7.6
| 7.6
| 0.77
| 0.77
| Model Providers
|-
|-
| '''[[DeepSeek-V2]]'''
| '''[[DeepSeek-V2]]'''
| DeepSeek logo
| [[DeepSeek]]
| 128k
| 128k
| –
| –
| –
| $0.17
| $0.17
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[Arctic]]'''
| '''[[Arctic]]'''
| Snowflake logo
| [[Snowflake]]
| 4k
| 4k
| 51
| 51
| –
| $0.00
| $0.00
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[Qwen2.5 72B]]'''
| '''[[Qwen2.5 72B]]'''
| Alibaba logo
| [[Alibaba]]
| 131k
| 131k
| 77
| 77
| –
| $0.40
| $0.40
| –
| 67.6
| 67.6
| 0.53
| 0.53
| Model Providers
|-
|-
| '''[[Qwen2.5 Coder 32B]]'''
| '''[[Qwen2.5 Coder 32B]]'''
| Alibaba logo
| [[Alibaba]]
| 131k
| 131k
| 72
| 72
| –
| $0.80
| $0.80
| –
| 84.1
| 84.1
| 0.38
| 0.38
| Model Providers
|-
|-
| '''[[Qwen2 72B]]'''
| '''[[Qwen2 72B]]'''
| Alibaba logo
| [[Alibaba]]
| 131k
| 131k
| 72
| 72
| –
| $0.63
| $0.63
| –
| 45.7
| 45.7
| 0.30
| 0.30
| Model Providers
|-
|-
| '''[[QwQ 32B-Preview]]'''
| '''[[QwQ 32B-Preview]]'''
| Alibaba logo
| [[Alibaba]]
| 33k
| 33k
| 46
| 46
| –
| $0.26
| $0.26
| –
| 66.9
| 66.9
| 0.40
| 0.40
| Model Providers
|-
|-
| '''[[Yi-Large]]'''
| '''[[Yi-Large]]'''
| 01.AI logo
| [[01.AI]]
| 32k
| 32k
| 61
| 61
| –
| $3.00
| $3.00
| –
| 67.5
| 67.5
| 0.47
| 0.47
| Model Providers
|-
|-
| '''[[GPT-4 Turbo]]'''
| '''[[GPT-4 Turbo]]'''
| OpenAI logo
| [[OpenAI]]
| 128k
| 128k
| 75
| 75
| –
| $15.00
| $15.00
| –
| 43.8
| 43.8
| 1.19
| 1.19
| Model Providers
|-
|-
| '''[[GPT-4]]'''
| '''[[GPT-4]]'''
| OpenAI logo
| [[OpenAI]]
| 8k
| 8k
| –
| –
| –
| $37.50
| $37.50
| –
| 30.5
| 30.5
| 0.73
| 0.73
| Model Providers
|-
|-
| '''[[Llama 3 70B]]'''
| '''[[Llama 3 70B]]'''
| Meta logo
| [[Meta]]
| 8k
| 8k
| 48
| 48
| –
| $0.89
| $0.89
| –
| 48.9
| 48.9
| 0.38
| 0.38
| Model Providers
|-
|-
| '''[[Llama 3 8B]]'''
| '''[[Llama 3 8B]]'''
| Meta logo
| [[Meta]]
| 8k
| 8k
| 45
| 45
| –
| $0.15
| $0.15
| –
| 118.9
| 118.9
| 0.34
| 0.34
| Model Providers
|-
|-
| '''[[Llama 2 Chat 70B]]'''
| '''[[Llama 2 Chat 70B]]'''
| Meta logo
| [[Meta]]
| 4k
| 4k
| –
| –
| –
| $1.85
| $1.85
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[Llama 2 Chat 13B]]'''
| '''[[Llama 2 Chat 13B]]'''
| Meta logo
| [[Meta]]
| 4k
| 4k
| –
| –
| –
| $0.00
| $0.00
| –
| –
| –
| –
| –
| Model Providers
|-
|-
| '''[[Llama 2 Chat 7B]]'''
| '''[[Llama 2 Chat 7B]]'''
| Meta logo
| [[Meta]]
| 4k
| 4k
| –
| –
| –
| $0.33
| $0.33
| –
| 123.8
| 123.8
| 0.37
| 0.37
| Model Providers
|-
|-
| '''[[Gemini 1.0 Pro]]'''
| '''[[Gemini 1.0 Pro]]'''
| Google logo
| [[Google]]
| 33k
| 33k
| –
| –
| –
| $0.75
| $0.75
| –
| 103.0
| 103.0
| 1.28
| 1.28
| Model Providers
|-
|-
| '''[[Claude 3 Sonnet]]'''
| '''[[Claude 3 Sonnet]]'''
| Anthropic logo
| [[Anthropic]]
| 200k
| 200k
| 57
| 57
| –
| $6.00
| $6.00
| –
| 67.9
| 67.9
| 0.74
| 0.74
| Model Providers
|-
|-
| '''[[Claude 2.1]]'''
| '''[[Claude 2.1]]'''
| Anthropic logo
| [[Anthropic]]
| 200k
| 200k
| –
| –
| –
| $12.00
| $12.00
| –
| 20.3
| 20.3
| 1.41
| 1.41
| Model Providers
|-
|-
| '''[[Claude 2.0]]'''
| '''[[Claude 2.0]]'''
| Anthropic logo
| [[Anthropic]]
| 100k
| 100k
| –
| –
| –
| $12.00
| $12.00
| –
| 29.9
| 29.9
| 0.81
| 0.81
| Model Providers
|-
|-
| '''[[Mistral Small (Feb '24)]]'''
| '''[[Mistral Small (Feb '24)]]'''
| Mistral logo
| [[Mistral]]
| 33k
| 33k
| 59
| 59
| –
| $1.50
| $1.50
| –
| 53.7
| 53.7
| 0.38
| 0.38
| Model Providers
|-
|-
| '''[[Mistral Large (Feb '24)]]'''
| '''[[Mistral Large (Feb '24)]]'''
| Mistral logo
| [[Mistral]]
| 33k
| 33k
| 56
| 56
| –
| $6.00
| $6.00
| –
| 38.9
| 38.9
| 0.43
| 0.43
| Model Providers
|-
|-
| '''[[Mistral 7B]]'''
| '''[[Mistral 7B]]'''
| Mistral logo
| [[Mistral]]
| 8k
| 8k
| 28
| 28
| –
| $0.16
| $0.16
| –
| 113.1
| 113.1
| 0.26
| 0.26
| Model Providers
|-
|-
| '''[[Mistral Medium]]'''
| '''[[Mistral Medium]]'''
| Mistral logo
| [[Mistral]]
| 33k
| 33k
| –
| –
| –
| $4.09
| $4.09
| –
| 44.5
| 44.5
| 0.38
| 0.38
| Model Providers
|-
|-
| '''[[Codestral]]'''
| '''[[Codestral]]'''
| Mistral logo
| [[Mistral]]
| 33k
| 33k
| –
| –
| –
| $0.30
| $0.30
| –
| 84.9
| 84.9
| 0.28
| 0.28
| Model Providers
|-
|-
| '''[[OpenChat 3.5]]'''
| '''[[OpenChat 3.5]]'''
| OpenChat logo
| [[OpenChat]]
| 8k
| 8k
| 44
| 44
| –
| $0.06
| $0.06
| –
| 73.4
| 73.4
| 0.30
| 0.30
| Model Providers
|-
|-
| '''[[Jamba Instruct]]'''
| '''[[Jamba Instruct]]'''
| AI21 Labs logo
| [[AI21 Labs]]
| 256k
| 256k
| –
| –
| –
| $0.55
| $0.55
| –
| 77.3
| 77.3
| 0.52
| 0.52
| Model Providers
|}
|}


Revision as of 19:32, 13 January 2025

Compare different large language models (LLMs): Concise comparison, detailed comparison, terminology definitions.

Concised Comparison

Model Creator Context Window Quality Index Blended Median Tokens/s Median First Chunk (s)
o1-preview OpenAI 128k 86 $27.56 144.5 21.12
o1-mini OpenAI 128k 84 $5.25 213.2 11.64
GPT-4o (Aug '24) OpenAI 128k 78 $4.38 85.6 0.65
GPT-4o (May '24) OpenAI 128k 78 $7.50 106.6 0.65
GPT-4o mini OpenAI 128k 73 $0.26 121.8 0.65
GPT-4o (Nov '24) OpenAI 128k 73 $4.38 115.7 0.38
GPT-4o mini Realtime (Dec '24) OpenAI 128k – $0.00 – –
GPT-4o Realtime (Dec '24) OpenAI 128k – $0.00 – –
Llama 3.3 70B Meta 128k 74 $0.69 71.3 0.50
Llama 3.1 405B Meta 128k 74 $3.50 30.2 0.72
Llama 3.1 70B Meta 128k 68 $0.72 72.7 0.44
Llama 3.2 90B (Vision) Meta 128k 68 $0.81 48.5 0.34
Llama 3.2 11B (Vision) Meta 128k 54 $0.18 131.0 0.29
Llama 3.1 8B Meta 128k 54 $0.10 184.7 0.34
Llama 3.2 3B Meta 128k 49 $0.06 202.1 0.38
Llama 3.2 1B Meta 128k 26 $0.04 468.9 0.38
Gemini 2.0 Flash (exp) Google 1m 82 $0.00 169.0 0.48
Gemini 1.5 Pro (Sep) Google 2m 80 $2.19 60.9 0.74
Gemini 1.5 Flash (Sep) Google 1m 72 $0.13 188.6 0.25
Gemma 2 27B Google 8k 61 $0.26 58.9 0.51
Gemma 2 9B Google 8k 55 $0.12 168.9 0.37
Gemini 1.5 Flash (May) Google 1m – $0.13 310.4 0.29
Gemini Experimental (Nov) Google 2m – $0.00 53.9 1.11
Gemini 1.5 Pro (May) Google 2m – $2.19 66.9 0.50
Gemini 1.5 Flash-8B Google 1m – $0.07 278.8 0.39
Claude 3.5 Sonnet (Oct) Anthropic 200k 80 $6.00 71.9 0.98
Claude 3.5 Sonnet (June) Anthropic 200k 76 $6.00 61.4 0.87
Claude 3 Opus Anthropic 200k 70 $30.00 25.9 2.00
Claude 3.5 Haiku Anthropic 200k 68 $1.60 65.1 0.71
Claude 3 Haiku Anthropic 200k 55 $0.50 122.7 0.72
Pixtral Large Mistral 128k 74 $3.00 36.7 0.39
Mistral Large 2 (Jul '24) Mistral 128k 74 $3.00 31.2 0.50
Mistral Large 2 (Nov '24) Mistral 128k 74 $3.00 37.4 0.52
Mistral Small (Sep '24) Mistral 33k 61 $0.30 61.5 0.32
Mixtral 8x22B Mistral 65k 61 $1.20 85.5 0.57
Pixtral 12B Mistral 128k 56 $0.13 70.5 0.37
Ministral 8B Mistral 128k 56 $0.10 135.8 0.29
Mistral NeMo Mistral 128k 54 $0.09 123.7 0.48
Ministral 3B Mistral 128k 53 $0.04 168.1 0.29
Mixtral 8x7B Mistral 33k 41 $0.50 110.1 0.36
Codestral-Mamba Mistral 256k 33 $0.25 95.8 0.44
Command-R+ Cohere 128k 55 $5.19 50.7 0.47
Command-R+ (Apr '24) Cohere 128k 45 $6.00 49.3 0.51
Command-R (Mar '24) Cohere 128k 36 $0.75 108.3 0.36
Aya Expanse 8B Cohere 8k – $0.75 165.6 0.16
Command-R Cohere 128k – $0.51 111.7 0.32
Aya Expanse 32B Cohere 128k – $0.75 120.3 0.18
Sonar 3.1 Small Perplexity 127k – $0.20 203.8 0.35
Sonar 3.1 Large Perplexity 127k – $1.00 57.8 0.31
Grok Beta xAI 128k 72 $7.50 66.5 0.42
Nova Pro Amazon 300k 75 $1.40 91.1 0.38
Nova Lite Amazon 300k 70 $0.10 147.7 0.34
Nova Micro Amazon 130k 66 $0.06 195.3 0.33
Phi-4 Microsoft Azure 16k 77 $0.09 85.1 0.21
Phi-3 Mini Microsoft Azure 4k – $0.00 – –
Phi-3 Medium 14B Microsoft Azure 128k – $0.30 50.7 0.43
Solar Mini Upstage 4k 47 $0.15 89.3 1.12
DBRX Databricks 33k 46 $1.16 74.2 0.43
Llama 3.1 Nemotron 70B NVIDIA 128k 72 $0.27 48.3 0.57
Reka Flash Reka AI 128k 59 $0.35 – –
Reka Core Reka AI 128k 58 $2.00 – –
Reka Flash (Feb '24) Reka AI 128k 46 $0.35 – –
Reka Edge Reka AI 128k 31 $0.10 – –
Jamba 1.5 Large AI21 Labs 256k 64 $3.50 50.8 0.71
Jamba 1.5 Mini AI21 Labs 256k – $0.25 83.5 0.48
DeepSeek V3 DeepSeek 128k 80 $0.90 21.1 0.95
DeepSeek-V2.5 (Dec '24) DeepSeek 128k 72 $0.17 63.8 1.15
DeepSeek-Coder-V2 DeepSeek 128k 71 $0.17 63.8 1.12
DeepSeek-V2.5 DeepSeek 128k – $1.09 7.6 0.77
DeepSeek-V2 DeepSeek 128k – $0.17 – –
Arctic Snowflake 4k 51 $0.00 – –
Qwen2.5 72B Alibaba 131k 77 $0.40 67.6 0.53
Qwen2.5 Coder 32B Alibaba 131k 72 $0.80 84.1 0.38
Qwen2 72B Alibaba 131k 72 $0.63 45.7 0.30
QwQ 32B-Preview Alibaba 33k 46 $0.26 66.9 0.40
Yi-Large 01.AI 32k 61 $3.00 67.5 0.47
GPT-4 Turbo OpenAI 128k 75 $15.00 43.8 1.19
GPT-4 OpenAI 8k – $37.50 30.5 0.73
Llama 3 70B Meta 8k 48 $0.89 48.9 0.38
Llama 3 8B Meta 8k 45 $0.15 118.9 0.34
Llama 2 Chat 70B Meta 4k – $1.85 – –
Llama 2 Chat 13B Meta 4k – $0.00 – –
Llama 2 Chat 7B Meta 4k – $0.33 123.8 0.37
Gemini 1.0 Pro Google 33k – $0.75 103.0 1.28
Claude 3 Sonnet Anthropic 200k 57 $6.00 67.9 0.74
Claude 2.1 Anthropic 200k – $12.00 20.3 1.41
Claude 2.0 Anthropic 100k – $12.00 29.9 0.81
Mistral Small (Feb '24) Mistral 33k 59 $1.50 53.7 0.38
Mistral Large (Feb '24) Mistral 33k 56 $6.00 38.9 0.43
Mistral 7B Mistral 8k 28 $0.16 113.1 0.26
Mistral Medium Mistral 33k – $4.09 44.5 0.38
Codestral Mistral 33k – $0.30 84.9 0.28
OpenChat 3.5 OpenChat 8k 44 $0.06 73.4 0.30
Jamba Instruct AI21 Labs 256k – $0.55 77.3 0.52

Detailed Comparison

Model Creator License Context Window Quality Index Normalized avg Chatbot Arena MMLU GPQA MATH-500 HumanEval Blended USD/1M Tokens Input Price Output Price Median Tokens/s P5 Tokens/s P25 Tokens/s P75 Tokens/s P95 Tokens/s Median First Chunk (s) P5 First Chunk (s) P25 First Chunk (s) P75 First Chunk (s) P95 First Chunk (s) Further Analysis
o1-preview OpenAI logo Proprietary 128k 86 1334 0.91 0.67 0.92 0.96 $27.56 $15.75 $63.00 143.8 68.9 121.6 164.6 179.6 21.28 13.40 17.04 27.80 46.49 – – Model Providers
o1-mini OpenAI logo Proprietary 128k 84 1308 0.85 0.58 0.95 0.97 $5.25 $3.00 $12.00 213.6 84.0 154.8 238.0 299.4 11.75 2.44 9.40 14.43 24.03 – – Model Providers
GPT-4o (Aug '24) OpenAI logo Proprietary 128k 78 1337 0.89 0.51 0.80 0.93 – – $4.38 $2.50 $10.00 85.6 40.3 61.5 109.3 143.6 0.66 0.33 0.43 0.91 1.92 Model Providers
GPT-4o (May '24) OpenAI logo Proprietary 128k 78 1285 0.87 0.51 0.79 0.93 – – $7.50 $5.00 $15.00 106.8 53.2 82.2 126.8 142.5 0.65 0.32 0.43 0.73 1.22 Model Providers
GPT-4o mini OpenAI logo Proprietary 128k 73 1273 0.82 0.44 0.79 0.88 – – $0.26 $0.15 $0.60 121.8 50.7 74.1 179.4 206.5 0.65 0.30 0.39 0.77 0.92 Model Providers
GPT-4o (Nov '24) OpenAI logo Proprietary 128k 73 1361 0.86 0.39 0.74 0.93 – – $4.38 $2.50 $10.00 115.1 71.7 95.0 140.0 165.5 0.38 0.27 0.32 0.52 0.75 Model Providers
GPT-4o mini Realtime (Dec '24) OpenAI logo Proprietary 128k – – – – – – – – $0.00 $0.00 $0.00 – – – – – – – – – – Model Providers
GPT-4o Realtime (Dec '24) OpenAI logo Proprietary 128k – – – – – – – – $0.00 $0.00 $0.00 – – – – – – – – – – Model Providers
Llama 3.3 70B Meta logo Open 128k 74 – 0.86 0.49 0.76 0.86 – – $0.67 $0.59 $0.73 67.2 23.6 31.2 275.7 2046.5 0.51 0.23 0.36 0.72 1.48 Model Providers
Llama 3.1 405B Meta logo Open 128k 74 1266 0.87 0.50 0.70 0.87 – – $3.50 $3.50 $3.50 30.2 12.2 21.8 66.3 160.3 0.72 0.37 0.52 0.89 2.02 Model Providers
Llama 3.1 70B Meta logo Open 128k 68 1249 0.84 0.43 0.64 0.80 – – $0.70 $0.60 $0.80 72.6 27.9 42.2 218.5 1971.9 0.44 0.24 0.34 0.60 1.00 Model Providers
Llama 3.2 90B (Vision) Meta logo Open 128k 68 – 0.84 0.42 0.65 0.80 – – $0.81 $0.81 $0.81 48.6 30.4 35.9 66.6 272.3 0.34 0.18 0.26 0.47 0.61 Model Providers
Llama 3.2 11B (Vision) Meta logo Open 128k 54 – 0.71 0.25 0.50 0.69 – – $0.18 $0.18 $0.18 131.0 35.9 118.2 165.4 752.4 0.29 0.17 0.23 0.36 0.56 Model Providers
Llama 3.1 8B Meta logo Open 128k 54 1172 0.71 0.26 0.50 0.68 – – $0.10 $0.10 $0.10 182.8 49.5 118.8 468.1 2161.5 0.34 0.17 0.27 0.47 0.73 Model Providers
Llama 3.2 3B Meta logo Open 128k 49 1103 0.64 0.21 0.50 0.60 – – $0.06 $0.06 $0.06 202.2 42.4 144.0 543.6 1623.1 0.38 0.15 0.26 0.49 0.93 Model Providers
Llama 3.2 1B Meta logo Open 128k 26 1054 0.35 0.14 0.13 0.40 – – $0.04 $0.04 $0.04 315.4 179.4 261.4 2053.8 3394.4 0.44 0.20 0.30 0.51 0.74 Model Providers
Gemini 2.0 Flash (exp) Google logo Proprietary 1m 82 – 0.87 0.59 0.90 0.91 – – $0.00 $0.00 $0.00 168.8 162.4 165.5 171.4 174.6 0.48 0.41 0.45 0.53 0.73 Model Providers
Gemini 1.5 Pro (Sep) Google logo Proprietary 2m 80 1301 0.86 0.59 0.88 0.88 – – $2.19 $1.25 $5.00 60.9 55.3 59.0 63.7 66.1 0.74 0.36 0.39 0.80 0.91 Model Providers
Gemini 1.5 Flash (Sep) Google logo Proprietary 1m 72 1271 0.75 0.45 0.83 0.83 – – $0.13 $0.07 $0.30 188.5 172.8 182.0 194.7 204.0 0.25 0.19 0.21 0.44 0.49 Model Providers
Gemma 2 27B Google logo Open 8k 61 1219 0.77 0.39 0.54 0.76 – – $0.26 $0.17 $0.51 58.7 32.0 46.3 67.8 74.4 0.51 0.19 0.32 1.53 5.94 Model Providers
Gemma 2 9B Google logo Open 8k 55 1190 0.73 0.31 0.50 0.64 – – $0.12 $0.12 $0.15 168.9 53.4 114.8 187.3 673.7 0.37 0.22 0.27 0.50 0.65 Model Providers
Gemini 1.5 Flash (May) Google logo Proprietary 1m – 1227 0.79 0.39 0.55 – – – $0.13 $0.07 $0.30 310.0 276.8 297.5 325.0 350.4 0.30 0.23 0.27 0.33 0.39 Model Providers
Gemini Experimental (Nov) Google logo Proprietary 2m – 1365 – – – – – – $0.00 $0.00 $0.00 53.8 51.1 52.6 55.1 56.4 1.12 0.78 0.90 1.82 3.31 Model Providers
Gemini 1.5 Pro (May) Google logo Proprietary 2m – 1260 0.86 0.46 0.66 – – – $2.19 $1.25 $5.00 66.9 62.7 64.6 68.4 70.2 0.50 0.38 0.42 0.81 0.88 Model Providers
Gemini 1.5 Flash-8B Google logo Proprietary 1m – 1211 0.75 0.30 0.70 – – – $0.07 $0.04 $0.15 279.0 226.9 260.1 288.8 300.0 0.39 0.27 0.35 0.46 0.59 Model Providers
Claude 3.5 Sonnet (Oct) Anthropic logo Proprietary 200k 80 1282 0.89 0.58 0.76 0.96 – – $6.00 $3.00 $15.00 71.8 37.6 44.8 78.0 89.6 0.98 0.68 0.78 1.36 2.23 Model Providers
Claude 3.5 Sonnet (June) Anthropic logo Proprietary 200k 76 1268 0.88 0.56 0.71 0.90 – – $6.00 $3.00 $15.00 61.4 41.6 49.9 78.9 91.0 0.87 0.68 0.75 1.06 1.45 Model Providers
Claude 3 Opus Anthropic logo Proprietary 200k 70 1248 0.84 0.50 0.64 0.83 – – $30.00 $15.00 $75.00 25.9 20.8 24.0 28.2 30.2 2.01 1.31 1.54 3.09 3.78 Model Providers
Claude 3.5 Haiku Anthropic logo Proprietary 200k 68 – 0.81 0.37 0.67 0.87 – – $1.60 $0.80 $4.00 65.1 51.1 58.6 75.4 105.1 0.71 0.54 0.64 0.93 1.20 Model Providers
Claude 3 Haiku Anthropic logo Proprietary 200k 55 1179 0.71 0.33 0.39 0.77 – – $0.50 $0.25 $1.25 122.6 97.6 112.0 134.7 152.7 0.72 0.35 0.43 0.79 1.06 Model Providers
Pixtral Large Mistral logo Open 128k 74 – 0.85 0.52 0.71 0.88 – – $3.00 $2.00 $6.00 36.6 18.9 33.6 39.3 41.1 0.40 0.33 0.34 0.51 2.22 Model Providers
Mistral Large 2 (Jul '24) Mistral logo Open 128k 74 1251 0.85 0.48 0.71 0.91 – – $3.00 $2.00 $6.00 31.2 25.8 29.2 33.8 35.4 0.50 0.38 0.45 0.55 0.89 Model Providers
Mistral Large 2 (Nov '24) Mistral logo Open 128k 74 – 0.85 0.47 0.72 0.90 – – $3.00 $2.00 $6.00 37.4 32.2 36.0 41.0 66.3 0.52 0.33 0.45 0.56 0.73 Model Providers
Mistral Small (Sep '24) Mistral logo Open 33k 61 – 0.74 0.34 0.56 0.81 – – $0.30 $0.20 $0.60 62.0 50.6 57.0 68.2 77.1 0.32 0.23 0.25 0.52 1.50 Model Providers
Mixtral 8x22B Mistral logo Open 65k 61 1148 0.76 0.37 0.56 0.74 – – $1.20 $1.20 $1.20 85.6 57.2 69.9 96.6 102.5 0.57 0.26 0.34 0.65 0.87 Model Providers
Pixtral 12B Mistral logo Open 128k 56 – 0.69 0.30 0.46 0.79 – – $0.13 $0.13 $0.13 70.5 61.6 66.8 77.3 81.0 0.37 0.26 0.31 0.48 0.74 Model Providers
Ministral 8B Mistral logo Open 128k 56 1183 0.59 0.30 0.57 0.79 – – $0.10 $0.10 $0.10 135.9 121.0 131.3 138.5 141.5 0.29 0.23 0.25 0.34 0.60 Model Providers
Mistral NeMo Mistral logo Open 128k 54 – 0.66 0.33 0.44 0.71 – – $0.09 $0.06 $0.14 124.0 51.8 76.5 158.8 163.6 0.48 0.18 0.25 0.55 0.71 Model Providers
Ministral 3B Mistral logo Proprietary 128k 53 – 0.58 0.26 0.54 0.74 – – $0.04 $0.04 $0.04 168.3 147.8 163.9 173.1 177.7 0.29 0.22 0.26 0.34 0.55 Model Providers
Mixtral 8x7B Mistral logo Open 33k 41 1114 0.63 0.30 0.31 0.38 – – $0.50 $0.45 $0.50 110.1 56.9 89.9 149.1 550.3 0.36 0.21 0.29 0.48 0.69 Model Providers
Codestral-Mamba Mistral logo Open 256k 33 – 0.25 0.03 0.24 0.79 – – $0.25 $0.25 $0.25 95.8 89.5 92.8 96.9 98.3 0.44 0.38 0.41 0.48 0.68 Model Providers
Command-R+ Cohere logo Open 128k 55 1215 0.75 0.34 0.40 0.71 – – $5.19 $2.75 $12.50 50.7 45.4 47.5 73.9 79.7 0.47 0.23 0.27 0.51 0.64 Model Providers
Command-R+ (Apr '24) Cohere logo Open 128k 45 1190 0.68 0.24 0.27 0.62 – – $6.00 $3.00 $15.00 49.3 45.6 47.4 64.9 77.2 0.51 0.25 0.34 0.58 0.64 Model Providers
Command-R (Mar '24) Cohere logo Open 128k 36 1149 0.59 0.26 0.16 0.44 – – $0.75 $0.50 $1.50 108.2 75.9 80.3 167.6 178.4 0.36 0.15 0.25 0.44 0.50 Model Providers
Aya Expanse 8B Cohere logo Open 8k – – – – – – – – $0.75 $0.50 $1.50 165.6 157.5 162.0 169.3 173.7 0.16 0.12 0.14 0.21 0.32 Model Providers
Command-R Cohere logo Open 128k – 1179 0.67 0.27 – 0.70 – – $0.51 $0.33 $1.05 111.8 102.7 107.6 117.7 124.9 0.32 0.15 0.21 0.36 0.39 Model Providers
Aya Expanse 32B Cohere logo Open 128k – 1207 0.67 – – – – – $0.75 $0.50 $1.50 120.3 114.9 118.4 123.3 127.5 0.18 0.15 0.16 0.25 0.32 Model Providers
Sonar 3.1 Small Perplexity logo Proprietary 127k – – – – – – – – $0.20 $0.20 $0.20 203.8 182.9 201.0 205.5 206.9 0.35 0.29 0.30 0.37 0.47 Model Providers
Sonar 3.1 Large Perplexity logo Proprietary 127k – – – – – – – – $1.00 $1.00 $1.00 57.8 45.7 54.0 60.8 64.6 0.31 0.29 0.30 0.36 0.46 Model Providers
Grok Beta xAI logo Proprietary 128k 72 1289 0.85 0.43 0.73 0.87 – – $7.50 $5.00 $15.00 66.5 56.9 64.4 67.6 68.8 0.42 0.34 0.38 0.47 0.56 Model Providers
Nova Pro Amazon logo Proprietary 300k 75 – 0.84 0.48 0.79 0.88 – – $1.40 $0.80 $3.20 91.3 77.6 82.4 96.4 102.5 0.38 0.35 0.37 0.39 0.42 Model Providers
Nova Lite Amazon logo Proprietary 300k 70 – 0.79 0.43 0.75 0.84 – – $0.10 $0.06 $0.24 148.0 126.4 134.8 156.7 165.5 0.33 0.30 0.32 0.35 0.38 Model Providers
Nova Micro Amazon logo Proprietary 130k 66 – 0.76 0.38 0.69 0.80 – – $0.06 $0.04 $0.14 195.8 170.9 186.0 208.3 219.5 0.33 0.30 0.32 0.35 0.39 Model Providers
Phi-4 Microsoft Azure logo Open 16k 77 – 0.85 0.57 0.81 0.87 – – $0.09 $0.07 $0.14 85.1 76.2 82.0 85.4 85.6 0.21 0.16 0.18 0.23 0.25 Model Providers
Phi-3 Mini Microsoft Azure logo Open 4k – 1037 – – – – – – $0.00 $0.00 $0.00 – – – – – – – – – – Model Providers
Phi-3 Medium 14B Microsoft Azure logo Open 128k – 1123 – – – – – – $0.30 $0.17 $0.68 50.8 17.6 44.9 52.7 54.2 0.43 0.38 0.42 0.45 0.49 Model Providers
Solar Mini Upstage logo Open 4k 47 1062 0.66 0.28 0.33 0.59 – – $0.15 $0.15 $0.15 89.1 84.3 87.7 90.5 92.5 1.12 1.07 1.12 1.14 1.42 Model Providers
DBRX Databricks logo Open 33k 46 1103 0.70 0.31 0.28 0.54 – – $1.16 $0.97 $1.73 74.2 50.2 68.0 82.9 83.1 0.44 0.27 0.32 0.51 0.71 Model Providers
Llama 3.1 Nemotron 70B NVIDIA logo Open 128k 72 1269 0.86 0.48 0.73 0.81 – – $0.27 $0.23 $0.40 48.3 27.4 44.3 69.8 71.0 0.57 0.23 0.32 0.64 0.79 Model Providers
Reka Flash Reka AI logo Proprietary 128k 59 – 0.73 0.34 0.53 0.74 – – $0.35 $0.20 $0.80 – – – – – – – – – – Model Providers
Reka Core Reka AI logo Proprietary 128k 58 1199 0.76 0.28 0.56 0.73 – – $2.00 $2.00 $2.00 – – – – – – – – – – Model Providers
Reka Flash (Feb '24) Reka AI logo Proprietary 128k 46 1165 0.65 0.27 0.33 0.61 – – $0.35 $0.20 $0.80 – – – – – – – – – – Model Providers
Reka Edge Reka AI logo Proprietary 128k 31 – 0.44 0.19 0.22 0.41 – – $0.10 $0.10 $0.10 – – – – – – – – – – Model Providers
Jamba 1.5 Large AI21 Labs logo Open 256k 64 1221 0.80 0.41 0.61 0.74 – – $3.50 $2.00 $8.00 50.8 43.9 49.9 51.8 58.4 0.72 0.49 0.68 0.74 0.81 Model Providers
Jamba 1.5 Mini AI21 Labs logo Open 256k – 1176 0.63 0.26 0.61 – – – $0.25 $0.20 $0.40 83.5 78.0 82.0 164.1 190.3 0.48 0.29 0.44 0.51 0.55 Model Providers
DeepSeek V3 DeepSeek logo Open 128k 80 – 0.87 0.56 0.85 0.92 – – $0.90 $0.90 $1.10 21.1 7.9 10.8 48.3 73.1 0.95 0.52 0.67 1.24 10.32 Model Providers
DeepSeek-V2.5 (Dec '24) DeepSeek logo Open 128k 72 – 0.81 0.42 0.76 0.88 – – $0.17 $0.14 $0.28 64.8 52.7 57.3 70.9 81.1 1.16 0.90 1.02 1.37 1.87 Model Providers
DeepSeek-Coder-V2 DeepSeek logo Open 128k 71 1178 0.80 0.42 0.74 0.87 – – $0.17 $0.14 $0.28 64.4 51.8 57.3 71.4 81.1 1.12 0.84 0.99 1.27 1.71 Model Providers
DeepSeek-V2.5 DeepSeek logo Open 128k – 1258 0.81 0.42 – 0.87 – – $1.09 $1.07 $1.14 7.6 6.9 7.2 8.0 8.2 0.77 0.60 0.70 0.88 17.15 Model Providers
DeepSeek-V2 DeepSeek logo Open 128k – 1220 0.80 0.42 – 0.87 – – $0.17 $0.14 $0.28 – – – – – – – – – – Model Providers
Arctic Snowflake logo Open 4k 51 1090 0.63 0.26 – 0.75 – – $0.00 $0.00 $0.00 – – – – – – – – – – Model Providers
Qwen2.5 72B Alibaba logo Open 131k 77 1259 0.86 0.50 0.84 0.89 – – $0.40 $0.40 $0.75 65.6 18.5 39.6 80.8 242.6 0.54 0.25 0.38 0.68 2.57 Model Providers
Qwen2.5 Coder 32B Alibaba logo Open 131k 72 1220 0.79 0.41 0.76 0.91 – – $0.80 $0.80 $0.80 84.1 34.7 45.4 102.8 344.0 0.37 0.21 0.28 0.51 1.25 Model Providers
Qwen2 72B Alibaba logo Open 131k 72 1187 0.83 0.40 0.77 0.86 – – $0.63 $0.63 $0.65 45.7 32.8 37.7 64.2 67.0 0.30 0.23 0.27 0.36 0.62 Model Providers
QwQ 32B-Preview Alibaba logo Open 33k 46 – 0.07 0.01 0.91 0.85 – – $0.26 $0.20 $0.60 66.3 35.0 52.1 105.2 329.9 0.40 0.21 0.32 0.58 2.05 Model Providers
Yi-Large 01.AI logo Proprietary 32k 61 1213 0.78 0.33 0.56 0.77 – – $3.00 $3.00 $3.00 67.4 63.1 64.5 78.9 81.8 0.47 0.35 0.40 0.78 1.64 Model Providers
GPT-4 Turbo OpenAI logo Proprietary 128k 75 1256 0.87 0.50 0.74 0.92 – – $15.00 $10.00 $30.00 43.8 22.7 35.3 52.8 58.7 1.19 0.51 0.63 1.44 2.02 Model Providers
GPT-4 OpenAI logo Proprietary 8k – 1186 – – – – – – $37.50 $30.00 $60.00 30.5 14.2 20.2 37.9 44.9 0.73 0.50 0.57 0.91 1.30 Model Providers
Llama 3 70B Meta logo Open 8k 48 1206 0.79 0.39 0.53 0.19 – – $0.88 $0.80 $0.88 48.2 19.3 32.9 130.8 349.0 0.40 0.23 0.30 0.55 1.36 Model Providers
Llama 3 8B Meta logo Open 8k 45 1152 0.64 0.30 0.32 0.53 – – $0.10 $0.07 $0.20 109.4 63.1 74.0 202.4 1203.6 0.35 0.19 0.30 0.41 0.73 Model Providers
Llama 2 Chat 70B Meta logo Open 4k – 1093 – – – – – – $1.85 $1.75 $2.17 – – – – – – – – – – Model Providers
Llama 2 Chat 13B Meta logo Open 4k – 1063 – – – – – – $0.00 $0.00 $0.00 – – – – – – – – – – Model Providers
Llama 2 Chat 7B Meta logo Open 4k – 1037 – – – – – – $0.33 $0.29 $0.46 123.8 119.1 122.3 126.0 130.1 0.37 0.34 0.35 0.40 0.55 Model Providers
Gemini 1.0 Pro Google logo Proprietary 33k – 1111 – – – – – – $0.75 $0.50 $1.50 103.1 97.1 98.7 105.2 107.5 1.28 1.20 1.24 1.31 1.39 Model Providers
Claude 3 Sonnet Anthropic logo Proprietary 200k 57 1201 0.77 0.37 0.41 0.71 – – $6.00 $3.00 $15.00 67.9 35.2 58.5 76.6 89.7 0.75 0.61 0.65 0.86 1.46 Model Providers
Claude 2.1 Anthropic logo Proprietary 200k – 1118 – – – – – – $12.00 $8.00 $24.00 20.3 13.0 13.5 29.4 31.0 1.41 0.79 0.82 1.78 2.01 Model Providers
Claude 2.0 Anthropic logo Proprietary 100k – 1132 – – – – – – $12.00 $8.00 $24.00 29.9 28.7 29.4 30.5 32.6 0.81 0.78 0.80 0.83 0.95 Model Providers
Mistral Small (Feb '24) Mistral logo Proprietary 33k 59 – 0.69 0.31 0.56 0.79 – – $1.50 $1.00 $3.00 53.7 48.8 52.1 61.6 73.2 0.38 0.25 0.33 0.41 0.62 Model Providers
Mistral Large (Feb '24) Mistral logo Proprietary 33k 56 1157 0.69 0.36 0.49 0.70 – – $6.00 $4.00 $12.00 38.8 29.8 36.8 42.5 45.4 0.43 0.34 0.40 0.52 0.98 Model Providers
Mistral 7B Mistral logo Open 8k 28 1008 0.33 0.19 0.16 0.42 – – $0.12 $0.11 $0.14 101.4 76.1 92.8 127.4 156.1 0.30 0.15 0.22 0.35 0.96 Model Providers
Mistral Medium Mistral logo Proprietary 33k – 1148 – – – – – – $4.09 $2.75 $8.10 44.5 40.3 42.7 45.3 46.8 0.38 0.31 0.33 0.45 17.22 Model Providers
Codestral Mistral logo Open 33k – – 0.23 – 0.80 – – – $0.30 $0.20 $0.60 84.8 78.1 82.1 86.5 88.8 0.28 0.24 0.26 0.31 0.49 Model Providers
OpenChat 3.5 OpenChat logo Open 8k 44 1076 0.56 0.22 0.31 0.68 – – $0.06 $0.06 $0.06 73.3 66.3 69.3 76.3 80.3 0.30 0.24 0.27 0.32 0.37 Model Providers
Jamba Instruct AI21 Labs logo Proprietary 256k – – 0.58 0.25 – – – – $0.55 $0.50 $0.70 77.1 70.4 74.3 169.6 193.7 0.52 0.29 0.45 0.54 0.58 Model Providers

Terms

  • Artificial Analysis Quality Index: Average result across our evaluations covering different dimensions of model intelligence. Currently includes MMLU, GPQA, Math & HumanEval. OpenAI o1 model figures are preliminary and are based on figures stated by OpenAI. See methodology for more details.
  • Context window: Maximum number of combined input & output tokens. Output tokens commonly have a significantly lower limit (varied by model).
  • Output Speed: Tokens per second received while the model is generating tokens (ie. after first chunk has been received from the API for models which support streaming).
  • Latency: Time to first token of tokens received, in seconds, after API request sent. For models which do not support streaming, this represents time to receive the completion.
  • Price: Price per token, represented as USD per million Tokens. Price is a blend of Input & Output token prices (3:1 ratio).
  • Output Price: Price per token generated by the model (received from the API), represented as USD per million Tokens.
  • Input Price: Price per token included in the request/message sent to the API, represented as USD per million Tokens.
  • Time period: Metrics are 'live' and are based on the past 14 days of measurements, measurements are taken 8 times a day for single requests and 2 times per day for parallel requests.

References

Retrieved from "https://aiwiki.ai/index.php?title=LLM_Comparisons&oldid=13763"