Magnitude (inference engine)

Magnitude is an open-source inference engine for running open-weight language models on local hardware, developed by Magnitude AI Inc., a San Francisco startup from the Y Combinator Summer 2025 batch.^[1][2][4] Its defining feature is on-device kernel tuning: when a user downloads a model, Magnitude spends up to a minute testing different ways of splitting the computation on that specific chip and keeps the fastest configuration that does not change the model's output.^[1][6] It ships as a desktop app and CLI for macOS, Windows and Linux, runs on Apple Silicon, NVIDIA and AMD GPUs or on a CPU alone, pairs most catalog models with a speculative decoding drafter, and exposes OpenAI-compatible and Anthropic-compatible APIs for coding agents.^[1][11] The code is written in Rust and released under the Apache 2.0 license.^[1][5]

The engine was introduced in a Launch HN post on Hacker News on 30 September 2026 with a claim of being "up to 2x faster than llama.cpp".^[5] On 15 October 2026 co-founder Tom Greenwald announced it again with a new headline figure, "over 2x faster than Ollama", based on the company's own benchmark of 11 models.^[6][3] An independent re-test published on X the next day reproduced the speedup on Magnitude's own workload, which asks the model to repeat text from its prompt, but found Magnitude slower than Ollama, with each engine using its default drafter, when the same model had to write new text.^[15]

Overview

History

Magnitude AI Inc. lists Tom Greenwald as founder and CEO and Anders Lie as founder and CTO; its Y Combinator profile gives the founding year as 2025, the batch as Summer 2025 and the team size as two.^[4] Before the inference engine, the company's best-known project was an open-source, vision-first browser agent, whose GitHub repository the company created in March 2025.^[18] In the Launch HN post the founders wrote that they had built that agent "to 4k+ GH stars and 100k+ downloads" and turned to inference after finding that no engine suited their goal of running it on local models.^[5]

The current magnitudedev/magnitude repository was created on 12 June 2026.^[13] Releases before version 0.2.0 ran on a llama.cpp-based backend: the 0.2.0 release notes, dated 30 September 2026, say the release replaced "the llama.cpp-based inference engine with Magnitude's own engine which automatically optimizes itself for any hardware".^[7] Greenwald later told a user on X that Magnitude is "not built on llama.cpp".^[17]

The Launch HN post titled "Magnitude (YC S25) - Self-optimizing inference engine for agents" went up on 30 September 2026.^[5] The Japanese technology site GIGAZINE covered the launch on 1 October 2026, summarizing the team's llama.cpp benchmark and its plans for expert streaming.^[14] Patch releases followed through early October; version 0.2.9, published on 15 October, added the ability to connect Claude Code to a local model without a Claude login and, according to its notes, improved performance "through better kernel tuning and token packing".^[8] The same day Greenwald posted the "over 2x faster than Ollama" announcement, which by 19 October 2026 showed about 98,700 views on X.^[6] As of 19 October 2026 the GitHub repository had about 7,140 stars and 490 forks.^[13]

Design

On-device kernel tuning

Magnitude's documentation argues that llama.cpp, Ollama and LM Studio "ship one-size-fits-all kernels, because tuning each model for each chip by hand would take forever".^[1] Magnitude's kernels are written with tunable parameters. On download, the engine tries different settings (in the README's words, "how many GPU threads team up, how big a chunk each one takes, how far ahead to load data") and keeps the fastest combination that leaves the output unchanged.^[1] Greenwald said on X that the search is capped at one minute because the team found it converges in that time, and that a new quantization of a model, or an update that changes the kernels, triggers a fresh tuning run while unchanged combinations keep their old results.^[17]

The company writes its own kernels for the most popular open-weight families instead of supporting arbitrary models. Greenwald said this is why Magnitude does not load arbitrary models from Hugging Face, though he said the team plans to allow external models from supported families.^[17] As of 19 October 2026 the public catalog listed 19 models, including Qwen3.8 27B and Flash Next, Qwen3.6 35B-A3B, Gemma 4 from E2B to 31B, Muse Glimmer 30B, MiniCPM-V 4.7 35B-A3B and Liquid AI's LFM2.5 24B-A2B.^[12]

Engine architecture

The repository's inference workspace describes a Rust engine built on a component called Seismic. The engine handles model interpretation, generation, scheduling and state, while Seismic "owns numerical compilation, device resources, and execution on CPU, Metal, CUDA, and Vulkan".^[10] The architecture document describes Seismic as a language, compiler and runtime that build candidate implementations of each computation for a given device and select among them. It states that the target is "near-optimal selection" and that a good benchmark score does not by itself establish a near-optimality bound.^[10] The document also says an earlier design built on Ops and TileLang has been replaced by Seismic.^[10]

Hardware support

The compatibility document sets one floor per backend:^[9]

The CUDA table in that document lists GB10 (NVIDIA DGX Spark) at compute capability 12.1 and explains that kernels are compiled on the user's machine by a bundled NVRTC 12.9, so no CUDA toolkit is needed. Turing and Volta GPUs are excluded from the CUDA backend and served through Vulkan, and Jetson Thor and architectures newer than NVRTC 12.9's targets, such as Rubin, are listed as unsupported for now.^[9] The user documentation states that Magnitude does not ship a ROCm backend; AMD GPUs run through Vulkan, and Intel Macs run on the CPU only.^[11] Greenwald said Intel GPUs "should already work" through the Vulkan backend.^[17]

Speculative decoding and memory

Magnitude says each model ships with a drafter for speculative decoding, although on 19 October 2026 two of the 19 catalog models (LFM2.5 24B-A2B and MiniCPM5 1B) had none listed.^[1][12] The documentation names three drafter types: multi-token prediction (MTP), DFlash and DSpark.^[11] The 0.2.0 release notes also mention DFlash2 and say that speculative decoding with these drafters had become faster than plain decoding on Apple Silicon for Qwen3.6-35B-A3B at 65,000 tokens of context (10.7% faster, where it had been 7% slower) and faster on NVIDIA (38.8% over plain decoding, up from 28.7%).^[7]

For memory, Magnitude uses a quantized KV cache, which the company says "halves memory per agent".^[1] In the Launch HN post the founders described two further mechanisms. Magnitude reserves only enough memory for the model weights at startup and grows the heap as agent sessions lengthen, freeing it when agents stop. It also uses "hybrid paged attention", which borrows from SGLang so that concurrent sessions share prefix caches while keeping memory placement favorable to single-session speed.^[5] The post names FlashAttention, FlashInfer, TurboQuant and SGLang's RadixAttention among the ideas the team drew on.^[5]

Positioning

The Launch HN post groups other engines into three kinds: engines built for batched datacenter serving at the cost of single-session speed (vLLM, SGLang), engines designed for broad compatibility rather than specific hardware (llama.cpp, Ollama), and engines specialized for particular hardware or models.^[5] Asked how Magnitude compares with vLLM and SGLang, Greenwald said on 15 October that the team had not benchmarked them and that Magnitude is "built for single-stream decode on your own machine".^[17]

Agent integration and API

Magnitude serves an HTTP API on port 10100 whenever the desktop app or magnitude serve is running, with an OpenAI-compatible base path (/inference/v1) and an Anthropic-compatible one (/inference/anthropic).^[11] The app's Connections screen configures supported agents in one click. The README lists Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi and Cline.^[1] On Linux and macOS, magnitude server setup installs Magnitude as a service that starts at boot and can serve agents on other computers.^[1] The documentation says models load on demand when an agent sends a request, unload when idle or when memory is tight, and use speculative decoding and prompt caching automatically.^[11]

Performance claims

Launch HN: against llama.cpp

The 30 September post reported one model, Qwen3.6 35B-A3B at 4-bit with 64,000 tokens of context and no speculative decoding:^[5]

15 October: against Ollama

The second announcement compared Magnitude with Ollama v0.35.1 across 11 models, with speculative decoding on for both engines. Magnitude reports an average 1.32x faster prefill and 2.19x faster decode on a Mac M4 Pro against Ollama's MLX runner, and 1.34x faster prefill and 2.17x faster decode on a DGX Spark against Ollama's llama.cpp runner.^[3][6] The averages are arithmetic means of per-model speedups. The per-model results, all self-reported:^[3]

Methodology and the company's caveats

The benchmark page describes the workload as "one chat request with about 64k tokens of context and 256 output tokens: passages from Moby-Dick, with the model asked to repeat the latest one", using greedy sampling with thinking turned off.^[3] Each request ran in a fresh engine process with an empty prompt cache. Each result is the median of two requests, except some Ollama figures, which the page says are single requests from an earlier run of the same harness.^[3] The models were "about 4-bit" on both engines: Ollama used its MLX NVFP4 tags on the Mac (the Qwen 3.5 4B row lists a GGUF Q4_K_M tag) and GGUF Q4_K_M tags on the DGX Spark.^[3]

The page lists several caveats of its own:^[3]

• On the Mac, Ollama's MLX runner drafted very few tokens for Gemma 4 E4B, 26B and 31B at 64k context even with its drafter loaded, so "those decode gains partly compare speculative decoding against almost none."
• Magnitude was slower in three cases, all included in the averages: Nemotron 3.5 decode on DGX Spark, Gemma 4 E2B prefill on the Mac and Gemma 4 26B prefill on DGX Spark.
• "The engines don't run identical weights, templates, or numerics, so outputs differ." Every request produced valid text for all 256 tokens, but "Answer quality isn't scored."

The benchmark harness is in the repository's inference/ directory, and the page gives the command used to reproduce a run in its prose-repeat mode.^[3]

Independent testing

On 16 October 2026 Niklas Lenz (@niklaslenz_ai), who tests local AI setups on his own hardware, published a re-test of one of the 11 models on X.^[15][16] He rebuilt the copy test from Magnitude's open harness and ran Qwen3.6 35B-A3B on an ASUS Ascent GX10 (a GB10 system with 128 GB) with the GPU clock capped at 2,200 MHz. He then added a second round in which the model writes a new story after a similar long prompt. Both engines ran with their default drafters, greedy sampling, thinking off and 256 output tokens, and he reports the median of three runs.^[15]

Lenz wrote that "'over 2x' holds for their test" and that copying is "the best case for a drafter": Magnitude's DFlash drafter had 75% of its guesses accepted when copying and 43% on the story.^[15] He said Ollama 0.35.1, the version Magnitude benchmarked, gave the same picture (85.7 tok/s copying, 63.8 tok/s on the story). He also noted that the setups were not identical, as in Magnitude's test: Ollama ran its registry tag (Q4_K_M with built-in MTP), while Magnitude ran Unsloth's UD-Q4_K_XL GGUF with its DFlash drafter. With Magnitude's exact GGUF imported into Ollama, where no drafter was available, Magnitude was faster in both rounds (188.9 against 45.6 tok/s copying, 50.4 against 46.1 tok/s on the story).^[15]

His Magnitude copy result (188.9 tok/s) was lower than the 216 tok/s on Magnitude's page for this model. Lenz listed differences that could matter: Magnitude starts a fresh engine for each request, whereas he used a running server with a random nonce to avoid prompt-cache hits, and Ollama ran its default f16 KV cache while Magnitude uses a quantized one. Like Magnitude, he did not score the quality of the output.^[15]

Roadmap

In the 15 October announcement Greenwald listed three planned features: expert streaming from RAM or disk to run models larger than GPU memory, splitting one model across all of a machine's GPUs and CPU, and attention pruning "that skips low-impact math, for much faster prefill".^[6] The Launch HN post also described a planned kernel compiler that would choose kernel fusion and implementations automatically.^[5] Asked about DeepSeek models, Greenwald said the team planned to add support "very soon" and that it would be especially relevant once expert streaming is implemented.^[17]

See also

• Ollama
• llama.cpp
• Splash (inference engine)
• Speculative decoding
• KV-cache quantization
• Quantization

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13Magnitude, "magnitudedev/magnitude" README, GitHub. github.com/...magnitude (accessed 19 October 2026)
2. ^1 ^2Magnitude, "Run open models as fast as your hardware allows", magnitude.dev. magnitude.dev (accessed 19 October 2026)
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Magnitude, "Magnitude vs Ollama benchmarks". magnitude.dev/benchmarks (accessed 19 October 2026)
4. ^1 ^2 ^3 ^4Y Combinator, "Magnitude: Run open models as fast as your hardware allows", company profile. ycombinator.com/...magnitude (accessed 19 October 2026)
5. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Anders Lie and Tom Greenwald, "Launch HN: Magnitude (YC S25) - Self-optimizing inference engine for agents", Hacker News, 30 September 2026. news.ycombinator.com/item
6. ^1 ^2 ^3 ^4 ^5Tom Greenwald (@tomgreenwald), launch post on X, 15 October 2026. x.com/...2110718297255670071
7. ^1 ^2 ^3Magnitude, "@magnitudedev/cli@0.2.0" release notes, GitHub, 30 September 2026. github.com/...%40magnitudedev%2Fcli%400.2.0
8. ^1 ^2Magnitude, "@magnitudedev/cli@0.2.9" release notes, GitHub, 15 October 2026. github.com/...%40magnitudedev%2Fcli%400.2.9
9. ^1 ^2 ^3Magnitude, "Compatibility" (inference/docs/compatibility.md), GitHub. github.com/...compatibility.md
10. ^1 ^2 ^3Magnitude, "Inference" README and "Inference V4" and "Seismic" architecture overviews, GitHub. github.com/...inference
11. ^1 ^2 ^3 ^4 ^5 ^6Magnitude documentation: "Hardware", "Models" and "API overview". docs.magnitude.dev/hardware ; docs.magnitude.dev/models ; docs.magnitude.dev/...overview (accessed 19 October 2026)
12. ^1 ^2Magnitude, "Browse open models" catalog. magnitude.dev/models (accessed 19 October 2026)
13. ^1 ^2 ^3GitHub API, repository metadata for magnitudedev/magnitude (created 12 June 2026; language, stars and forks). api.github.com/...magnitude (accessed 19 October 2026)
14. ^GIGAZINE, "Magnitude, an AI agent inference engine that automatically optimizes for hardware to accelerate open models by up to 2x...", 1 October 2026. gigazine.net/...20261001-magnitude
15. ^1 ^2 ^3 ^4 ^5 ^6Niklas Lenz (@niklaslenz_ai), "Qwen3.6 35B-A3B on one DGX Spark: Magnitude copies text 2.17x faster than Ollama, but writes a new story at 0.79x Ollama's speed", X, 16 October 2026. x.com/...2111204284453044383
16. ^Niklas Lenz, niklaslenz.com, "Local AI, measured at home". niklaslenz.com (accessed 19 October 2026)
17. ^1 ^2 ^3 ^4 ^5 ^6Tom Greenwald (@tomgreenwald), replies on X, 15 October 2026: on tuning time x.com/...2110794895120097472 ; on retuning x.com/...2110728331930784054 ; on llama.cpp x.com/...2110796784180335071 ; on vLLM and SGLang x.com/...2110815941164421276 ; on the model catalog x.com/...2110827076089765943 ; on Intel GPUs x.com/...2110815129159757900 ; on DeepSeek x.com/...2110788259928506562
18. ^Magnitude, "magnitudedev/browser-agent: Open-source, vision-first browser agent", GitHub. github.com/...browser-agent