AI Wiki

InferenceX

AI BenchmarksAI HardwareAI Inference
17 min read
Updated
Suggest editHistoryTalk22 citationsFact-checked
RawGraph

InferenceX, launched in October 2025 under the name InferenceMAX, is an open-source benchmark that continuously measures large language model inference performance across AI accelerators and serving software. It is built and operated by SemiAnalysis, which describes itself as "an independent semiconductor and AI research firm," and is published as a free public dashboard backed by a public GitHub repository under the Apache 2.0 license [1][2]. Its organizing idea is that inference benchmarks decay quickly because the software stack moves faster than the silicon, so the suite re-runs its recipes on real hardware whenever a configuration changes rather than publishing results in periodic submission rounds [2][3].

Launch as InferenceMAX

SemiAnalysis published the launch article on 9 October 2025, authored by Kimbo Chen, Dylan Patel and Daniel Nishball among others [1]. The stated motivation was that "benchmarks conducted at a fixed point in time quickly go stale and do not represent the performance that can be achieved with the latest software packages," since kernel work, scheduler changes and new releases of vLLM, SGLang and TensorRT-LLM can move measured performance substantially within weeks [1].

The v1 release covered NVIDIA's GB200 NVL72, B200, H200 and H100, and AMD Instinct MI355X, MI325X and MI300X [1]. Three models were benchmarked at launch: Llama 3.3 70B as a dense enterprise-style model, DeepSeek-R1 as a large sparse mixture-of-experts model, and gpt-oss 120B as a smaller MoE model [1]. Tom's Hardware reported on 10 October 2025 that both AMD and NVIDIA "were asked for real-world configuration sets for GPUs and the software stack, as these can be tuned thousands of different ways," a detail that matters for interpreting every number the suite produces [4].

Rename to InferenceX

The project was renamed InferenceX with the release of its second major version on 16 February 2026 [2]. SemiAnalysis has not published an explanation for the change. The v2 article, the repository README and the site all carry the name as "InferenceX (formerly InferenceMAX)," and the trademark symbol now attaches to InferenceX rather than InferenceMAX [2][3][5]. Older branding survives in places: several partner testimonials on the site still say InferenceMAX, and the original launch article retains its InferenceMAX title [1][6]. The earlier domains inferencemax.ai and inferencemax.semianalysis.com redirect to the current site, as does the shorter inferencex.com, which the repository README gives as the dashboard's public address [5]. The GitHub repository, originally at InferenceMAX/InferenceMAX, moved to SemiAnalysisAI/InferenceX [5].

As of August 2026 the current, correct name for the project is InferenceX. It is best described as InferenceX and treated as a continuous line of releases: InferenceMAX v1 in October 2025, InferenceX v2 in February 2026.

How the benchmark runs

InferenceX is organized around auditability rather than trust. SemiAnalysis describes a four-step chain in which every dashboard point traces back to a public workflow run [3]:

  1. Every combination of hardware, framework, model and precision is a shell script committed to the public repository, with the container image, command line and parallelism settings pinned in source.
  2. GitHub Actions schedules the workflow onto the actual target accelerator, using GPU servers registered as self-hosted runners, and streams the job log publicly while it executes.
  3. Request latencies, token counts, GPU power telemetry and evaluation samples are uploaded as run artifacts.
  4. Successful runs are ingested into the database and surfaced on the dashboard, where each chart tooltip links back to the run that produced the point.

GitHub Actions retains logs and artifacts for 90 days, so SemiAnalysis says it also publishes a weekly snapshot of the full benchmark database as a public GitHub Release for longer-term auditability [3]. In practice the releases run roughly weekly with occasional gaps. Those snapshots are published on the separate dashboard repository, SemiAnalysisAI/InferenceX-app, as compressed database dumps under tags of the form db-dump/<date> [7]. The benchmark repository itself is licensed Apache 2.0; the dashboard application repository is licensed GPL-3.0 per its LICENSE file, although the dashboard site's own article footer describes the application source as AGPL-3.0 [5][7].

The run cadence changed between versions. v1 ran a full nightly sweep of every configuration [1]. In v2, SemiAnalysis wrote that running every configuration each night had become "way too time consuming and wasteful" as chips, disaggregated prefill and wide expert parallelism were added, and that "benchmarks only really need to be re-run when recipes change or a new software version is released" [2]. Sweeps are now triggered by entries in a changelog at the root of the repository: a developer making a performance-affecting change lists the affected configuration keys, and a merged pull request fans those keys out from a master YAML configuration into individual GitHub Actions jobs [2]. The site describes the result as roughly 1,000 or more new benchmark data points per week on average [6]. The repository had approximately 1,300 stars and 240 forks as of August 2026 [5].

The throughput and interactivity tradeoff

The conceptual core of InferenceX is that a single number cannot describe inference performance, because throughput and responsiveness trade off against one another. Interactivity, measured in tokens per second per user, is how fast one user's tokens arrive, and is the inverse of time per output token. Throughput, measured in tokens per second per GPU, is how many tokens the system produces in aggregate [2]. Batching more requests together raises aggregate throughput but gives each individual request a smaller share of compute, so each user waits longer [2]. SemiAnalysis compares the choice to riding a bus versus a race car: the bus amortizes its cost across many riders but stops often, while the race car carries one or two people quickly at much higher cost per passenger [2].

Because of that tradeoff, InferenceX reports results as a curve rather than a point. For a given hardware and software configuration it sweeps concurrency levels and plots throughput against interactivity, then identifies the Pareto-optimal points where no other configuration is better on both axes at once [1]. Connecting those points produces the Pareto frontier that the dashboard displays. SemiAnalysis argues that maximum achievable throughput is misleading in isolation because it is normally reached only at one very low interactivity level, and that readers should locate their own workload on the curve: real-time speech needs low latency, a question-answering chatbot tolerates more [2].

Cost and power curves largely follow the same shape, because more tokens per hour amortize fixed dollar-per-hour and watt costs over more output [2].

Metrics

The metric set reported on the dashboard is [3]:

MetricUnit
Interactivitytokens per second per user
Token throughput per GPUtok/s/gpu (also split input and output)
Token throughput per provisioned megawatttok/s/MW
P99 time to first tokenmilliseconds
Cost per million tokensUSD, total, input and output
Energy per tokenjoules per token
Custom cost and power modelsuser-defined

Cost per million tokens is computed against several pricing regimes, including hyperscaler, neocloud and rental pricing, and users can substitute their own cost and power assumptions [3]. SemiAnalysis calls total cost of ownership per million tokens "the true north star that customers care about," on the argument that raw performance differences only become economically meaningful once normalized by what the hardware and its power cost [1]. Tokens per provisioned megawatt is expressed as a rate against utility power and is intended to include GPUs, CPUs, networking and facility overhead such as cooling rather than GPU board power alone [1].

v2 added a numerical accuracy check alongside the performance sweep, using GSM8K evaluations to catch cases where a throughput optimization silently degrades output quality [2].

Workloads, models and frameworks

Each model is tested across three sequence-length scenarios, chosen to stand in for common deployment shapes: chat at 1,024 input and 1,024 output tokens, reasoning at 1,024 input and 8,192 output, and summarization at 8,192 input and 1,024 output [1]. Input lengths are randomized between 80 and 100 percent of the nominal length to mimic request diversity [1].

The model set has grown well beyond the original three. As of August 2026 the site lists DeepSeek-R1-0528, gpt-oss-120b, Llama 3.3 70B Instruct FP8, Qwen3.5-397B-A17B from the Qwen family, Kimi K2.5, K2.6, K2.7-Code and K3, MiniMax M3 and earlier M2.5 and M2.7, GLM-5, GLM-5.1 and GLM-5.2, and DeepSeek V4 Pro [3].

On the software side the dashboard covers vLLM, SGLang, TensorRT-LLM and combinations built on NVIDIA Dynamo, plus llm-d with vLLM, Mooncake and AMD's ATOM and MoRI paths [3]. Precisions include FP4, FP8, BF16 and INT4, running on both CUDA and ROCm [3]. Disaggregated serving with separate prefill and decode pools, multi-token prediction, and wide expert parallelism for MoE models are all treated as first-class configuration axes rather than footnotes [2][3]. Many of the engines involved are built on PyTorch, and models are pulled directly from Hugging Face [6].

Hardware coverage

The distinction between what is actually benchmarked and what is announced matters, because the marketing copy and the FAQ do not always read the same way. The site's own accelerator list, and the "coming soon" language on the home page and repository README, give the following picture as of August 2026 [3][5][6]:

VendorAcceleratorStatus
NVIDIAH100, H200Benchmarked
NVIDIAB200, B300Benchmarked
NVIDIAGB200 NVL72, GB300 NVL72Benchmarked
NVIDIARTX 6000 ProBenchmarked
NVIDIAVR200 NVL72, Rubin NVL8Announced, not yet live
AMDMI300X, MI325X, MI355XBenchmarked
AMDMI455X UALoE72Announced, not yet live
GoogleTPU v7 IronwoodAnnounced, not yet live
AWSTrainium 2 and 3 (Trainium 3 named in the v2 article; Trainium 2 only in the repository description)Announced, not yet live

SemiAnalysis wrote in February 2026 that it was "actively working on adding both TPU and Trainium" [2]. Separately, a June 2026 SemiAnalysis article tracking DeepSeek V4 performance from day 0 to day 43 included analysis of Huawei Ascend 950DT alongside GB300 NVL72, MI355X and B200, although Huawei parts are not listed among the accelerators on the public dashboard [8].

Comparison with MLPerf and Artificial Analysis

InferenceX is frequently described in contrast to MLPerf Inference, run by MLCommons, and to Artificial Analysis. The three answer different questions.

InferenceXMLPerf InferenceArtificial Analysis
CadenceContinuous; re-runs when a recipe or software version changes [3]Submission rounds roughly every six months; v5.1 preceded v6.0 by six months [9]Ongoing measurement of live endpoints [10]
Who runs the testSemiAnalysis, on hardware it has access to [2][11]Each submitting organization runs on its own systems [9]Artificial Analysis, against public APIs [10]
Who tunesSemiAnalysis writes the recipes; AMD and NVIDIA supply recommended configurations [4]Submitters tune their own stack within division rules [12]Not applicable; the provider controls its own serving stack [10]
Rules structureOpen repository, no divisionsClosed division requires the reference model; Open division permits a different or retrained model [12]Standardized token accounting across providers [10]
Object of measurementAccelerators plus open-source serving softwareSystems submitted by vendors and integratorsCommercial inference endpoints and model quality

The sharpest structural difference is who performs the measurement. MLPerf results are produced by the submitters themselves and published in coordinated rounds, which gives vendors control over their own tuning but means results appear only a few times a year [9][12]. InferenceX inverts this: SemiAnalysis owns the recipes and the execution, publishes them continuously, and exposes the logs, but vendors still influence the outcome by recommending configurations and by shipping the software being measured [3][4]. Artificial Analysis is different again, measuring what customers actually experience from commercial serving endpoints rather than what a given accelerator can do; its documentation states that its results "are not intended to represent the maximum possible performance on any particular hardware platform" but rather "the real-world performance customers experience across providers" [10].

Supporters and vendor relationships

InferenceX publishes a long list of organizations and individuals under the label "supported and trusted by," which includes OpenAI, Microsoft, Meta, Oracle, MiniMax, Moonshot AI, Alibaba Qwen, Zhipu GLM, Tri Dao, the vLLM and SGLang projects, the PyTorch Foundation, CoreWeave, Nebius, TensorWave, WEKA, Stanford, Hugging Face, Lambda, UC San Diego, Red Hat, the White House and SambaNova [3][21]. These relationships are not uniform and the site does not describe them as funding. Most are public testimonials from named executives or maintainers, collected on a supporters page [21].

Two categories are materially different and are documented in the acknowledgements of the SemiAnalysis articles. The first is hardware access: SemiAnalysis thanked Jensen Huang and Ian Buck of NVIDIA for access to GB200 NVL72 and B200 systems at launch, and in v2 for access to GB300 NVL72 plus servers covering every NVIDIA SKU of the prior four years, described as close to 1,000 GPUs for a full sweep; it thanked Lisa Su and Anush Elangovan for MI355X and CDNA3 hardware [1][2]. The second is engineering support: named engineers at both companies helped validate and optimize configurations, and dozens of AMD engineers contributed both to InferenceX and to upstream vLLM and SGLang bug fixes [1][2]. Compute for the benchmark has come from Crusoe, CoreWeave, Nebius, TensorWave, Oracle and Together AI [1]. SambaNova's chief executive framed his company's involvement prospectively rather than as current participation, saying it looked forward to taking part with its SN50 part [21].

Results and industry reception

NVIDIA responded to the v1 results with a technical blog on 13 October 2025 claiming that Blackwell led the benchmark, citing figures including roughly 60,000 tokens per second per GPU on gpt-oss-120b with TensorRT-LLM and a DeepSeek-R1 cost per million tokens of $1.56 on H200 against just over $0.10 on GB200 NVL72 at roughly 75 tokens per second per user, which NVIDIA called a 15x reduction [13]. AMD published its own article on 10 October 2025 in which Anush Elangovan, VP of GPU software, said the nightly benchmarks "help show the speed of improvement of the AMD software stack," and which showed a roughly 5x generational gain from MI300X to MI355X on gpt-oss-120b [11]. Both the vLLM and SGLang projects published posts describing joint optimization work done around the launch [14][15].

The v2 results in February 2026 were more pointed. SemiAnalysis reported that with disaggregated prefill, wide expert parallelism and FP4 enabled together, NVIDIA's B200, B300 and rack-scale GB200 and GB300 NVL72 held a large lead, and that GB300 NVL72 reached up to 100x the performance of a strong H100 disaggregated baseline in an FP8-versus-FP4 comparison and 65x on FP8 versus FP8 [2]. It identified composability as AMD's central problem: individual optimizations worked well in isolation, but "when combined with other optimizations, the result is not as competitive as one would expect" [2]. The article was equally direct with NVIDIA, recommending that it staff more engineers onto the open SGLang and vLLM ecosystems rather than concentrating on TensorRT-LLM [2]. Trade coverage of v2 focused on the AMD software gap [16].

Criticism and disputes

The most substantive public dispute is not about InferenceX's data but about how NVIDIA presented it. In an article dated 18 March 2026, AMD wrote that at GTC 2026 NVIDIA had used InferenceX data to show GB300 NVL72 in FP4 with MTP delivering "50X higher tokens-per-watt and 35X lower cost-per-token than last-generation Hopper (FP8)," and objected that "each choice favors NVIDIA's result" [17]. AMD noted that NVIDIA's comparison used MTP=3 while AMD's default at the time was MTP=1, and used FP4 when FP8 remains a common production choice. Comparing the same operating modes, AMD argued, MI355X cost per token was "materially lower than GB300 NVL72 at high concurrency, 60+ TPS/user" on DeepSeek-R1 in FP8 without MTP, and it cited InferenceX itself as the source for those charts [17]. AMD's framing of the underlying benchmark was not hostile: it called InferenceX "an independent inference benchmarking framework" whose breadth "is the point," since "no single operating point can tell the whole story," and it argued the correct comparison is a customer running their own model, context length and latency target on both platforms [17]. AMD had made a related argument at the October 2025 launch, cautioning that each published result "represents a snapshot in time" that reflects "the current state of kernel efficiency for specific model configurations, rather than the inherent capability of the hardware" [11].

A second line of criticism concerns reproducibility in practice. In a March 2026 comparison written by a competing benchmarking vendor, Silicon Data, engineer Platon Slynko wrote that "InferenceX results come from hardware that SemiAnalysis operates internally" and that "there is no way for the public to run InferenceX on their own systems," and raised the possibility that vendor-supplied hardware may not represent what typical customers receive [18]. The critique came from a party with a commercial interest in the comparison and should be read accordingly. It also sits against a practical counterweight: AMD's October 2025 article published step-by-step instructions for forking the repository, registering self-hosted runners and launching a benchmark on one's own MI355X machines, which suggests the recipes are runnable by anyone with equivalent hardware [11].

SemiAnalysis has itself documented the suite's main methodological limits. As of the v2 article, InferenceX used "completely random tokens as input for benchmarking," which forces prefix caching to be disabled because the expected cache hit rate on random data is zero, and every conversation is single-turn [2]. The company acknowledged that this "provides a good baseline Pareto frontier" but "is not a practical benchmark setup that mimics real-world production inference workloads" [2]. It announced plans to move to real multi-turn datasets such as WildChat with prefix caching and KV cache CPU offloading enabled, and to add an agentic coding benchmark [2]. That work has since landed: the project now publishes replayable conversation traces captured from real Claude Code sessions, storing per-turn token counts and 64-token KV-cache block hashes rather than prompt text, released on Hugging Face under Apache 2.0 [19][20].

A structural question applies to the project: SemiAnalysis sells subscription research and consulting to many of the same companies whose hardware it benchmarks [22]. The company addresses this only indirectly, through the reproducibility chain: every point links to the public workflow run that produced it, and the recipes and raw artifacts are open [3].

References

  1. ^SemiAnalysis, "InferenceMAX: Open Source Inference Benchmarking," 9 October 2025. newsletter.semianalysis.com/...en-source-inference
  2. ^SemiAnalysis, "InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper (Formerly InferenceMAX)," 16 February 2026. newsletter.semianalysis.com/...nvidia-blackwell-vs
  3. ^InferenceX, "About," SemiAnalysis. inferencex.semianalysis.com/about
  4. ^Bruno Ferreira, "InferenceMax AI benchmark tests software stacks, efficiency, and TCO," Tom's Hardware, 10 October 2025. tomshardware.com/...-performance-changes-over-time
  5. ^SemiAnalysisAI/InferenceX repository, GitHub. github.com/...InferenceX
  6. ^InferenceX home page, SemiAnalysis. inferencex.semianalysis.com
  7. ^SemiAnalysisAI/InferenceX-app releases, GitHub. github.com/...releases
  8. ^SemiAnalysis, "DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time: GB300 NVL72, Huawei, MI355X, B200," 9 June 2026. newsletter.semianalysis.com/...-day-43-performance
  9. ^MLCommons, "MLCommons Releases New MLPerf Inference v6.0 Benchmark Results," 1 April 2026. mlcommons.org/...mlperf-inference-v6-0-results
  10. ^Artificial Analysis, "Methodology." artificialanalysis.ai/methodology
  11. ^AMD, "InferenceMAX: Benchmarking Progress in Real Time," 10 October 2025. amd.com/...emax-benchmarking-progress-in-real-time
  12. ^MLCommons, "MLPerf Inference: Datacenter." mlcommons.org/...inference-datacenter
  13. ^NVIDIA Technical Blog, "NVIDIA Blackwell Leads on SemiAnalysis InferenceMAX v1 Benchmarks," 13 October 2025. developer.nvidia.com/...is-inferencemax-benchmarks
  14. ^vLLM Blog, "SemiAnalysis InferenceMAX: vLLM and NVIDIA Accelerate Blackwell Inference," 9 October 2025. blog.vllm.ai/...blackwell-inferencemax
  15. ^LMSYS Org, "SGLang and NVIDIA Accelerating SemiAnalysis InferenceMAX and GB200 Together," 14 October 2025. lmsys.org/...2025-10-14-sa-inference-max
  16. ^Ben Wodecki, "Nvidia pulls ahead as AMD's software stack falls short: report," SDxCentral, 19 February 2026. sdxcentral.com/...oftware-stack-falls-short-report
  17. ^AMD, "The Many Aspects of Inference Performance," 18 March 2026. amd.com/...the-many-aspects-of-inference-performance
  18. ^Platon Slynko, "SiliconMark vs. InferenceX: Comparing AI Inference Benchmarking Frameworks," Silicon Data, 4 March 2026. silicondata.com/...siliconmark-vs.-inferencex
  19. ^InferenceX, "Agentic Benchmark Datasets," SemiAnalysis. inferencex.semianalysis.com/datasets
  20. ^semianalysisai datasets, Hugging Face. huggingface.co/semianalysisai
  21. ^InferenceX, "Supporters," SemiAnalysis. inferencex.semianalysis.com/quotes
  22. ^SemiAnalysis, "About," describing its subscription products and consulting engagements. semianalysis.com/about

Improve this article

Add missing citations, update stale details, or suggest a clearer explanation. Every suggestion is reviewed for sourcing before it goes live.

v1 · 3,342 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Reviewer note: Benchmarked versus announced hardware, framework list, run cadence, licensing and the February 2026 rename confirmed against the project's own dashboard, FAQ and GitHub repositories on 2026-08-01.

Suggest edit