Best Local and On-Device LLMs

As of July 2026, the best local LLM to run offline depends almost entirely on how much memory you can give it, so this guide ranks picks by hardware tier. The short answer: run Phi-4-mini or Gemma 4 E4B on an 8 GB machine, Gemma 4 12B on a 16 GB laptop, Qwen3.6-27B on a 24 GB GPU, and gpt-oss-120b or Llama 4 Scout once you have 48 GB or more. Every model here is open-weight, runs fully offline, and installs in one command through Ollama or LM Studio. If you want a single default for a typical laptop, use Gemma 4 12B at Q4_K_M: it needs about 7.6 GB, fits a mainstream 16 GB machine, and scores 77.2 on MMLU-Pro. ^[1][2][3]

This is a deployment guide for running models yourself. It ranks models by memory budget and pairs each tier with the right runtime. For the model class in the abstract, and why sub-10B models now rival much larger ones, see small language model.

Best local LLM by RAM tier: the short list

The table below gives one top pick per tier. "Footprint" is the on-disk size of the recommended quant, which is roughly the minimum memory the weights occupy; leave headroom for the KV cache, which grows with context length. As a rule of thumb, a Q4_K_M model needs about 0.55 to 0.7 GB of memory per billion parameters plus context. Last verified: July 2026.

Scroll sideways for more →

Sources: official model cards, Ollama library sizes, and benchmark leaderboards, detailed per tier below. ^{[1][3][4][5][6]}

Which local model should I run on 8 GB of RAM?

On an 8 GB laptop or phone-class device, the top pick is Gemma 4 E4B for all-round use and Phi-4-mini when you need reasoning or CPU-only inference.

Phi-4-mini is Microsoft's 3.8B Phi model, released February 26, 2025 under the MIT license. At Q4_K_M it occupies about 2.5 GB (INT4 drops to ~2.1 GB), leaves room for other apps, and scores 67.3 on MMLU and 52.8 on MMLU-Pro, ahead of most models in its 3B to 4B class. ^[1][5][12] It has the best CPU-only throughput in its class, roughly 12 tokens per second on a modern i7 with no GPU, and 15 to 20 tokens per second on an M1 MacBook Air, with a 128K context window. ^[5][12]

Gemma 4 E4B is the newer all-rounder from Google DeepMind, part of the Gemma 4 family released April 2, 2026 under Apache 2.0. It has 8B total but only 4.5B effective parameters thanks to Per-Layer Embeddings, runs in as little as 3 GB of memory at 4-bit, and adds native audio input plus image understanding and a 128K context, which Phi-4-mini lacks. ^[1][2] For phones and single-board computers, Gemma 4 E2B (2.3B effective) squeezes into under 1.5 GB. ^[1] Other solid 8 GB options are Llama 3.2 3B for general chat and Qwen3 4B for multilingual work. Gemma 4 E2B and E4B also succeed the Gemma 3n edge models, which remain fine picks if already installed.

Which local model is best for a 16 GB laptop?

For 16 GB, the top pick is Gemma 4 12B, with DeepSeek-R1-Distill for math and reasoning.

Gemma 4 12B is a unified multimodal model of 11.95B parameters with a 256K context. Its Q4_K_M build is a 7.6 GB download and needs about 6.6 GB of VRAM for weights, so it runs comfortably in 16 GB with context to spare. ^[3] It scores 77.2 on MMLU-Pro and 78.8 on GPQA Diamond, and runs around 21 tokens per second on an RTX 4060 and in real-time-chat territory on a 24 GB RTX 4090. ^[3][10] For pure reasoning and competition math, DeepSeek-R1-Distill-Qwen-14B is the standout: the Q4_K_M GGUF is 8.99 GB and fits in 12 GB, and it distills DeepSeek-R1's chain-of-thought into a 14B body (the newer DeepSeek-R1-0528-Qwen3-8B is a lighter alternative). ^[9] Qwen3 14B is the strongest general dense model at this size. If you prefer a fast mixture of experts design, Gemma 4 26B A4B activates only 3.8B of its 25.2B parameters per token, so it generates at roughly 4B speed while loading in about 15 GB at Q4. ^[1][2]

Best local LLM for a 24 GB GPU

With 24 GB (an RTX 3090 or 4090, or a 24 to 32 GB Apple Silicon Mac), the best overall model is Qwen3.6-27B, and gpt-oss-20b is the pick for raw speed and coding.

Qwen3.6-27B is a dense, Apache 2.0 model from Alibaba, released April 22, 2026 with a 256K context. At Q4_K_M it is about 17 GB and fits 24 GB with room for a long context. ^[3][6] It scores 86.2 on MMLU-Pro, 87.8 on GPQA Diamond, and 77.2 on SWE-bench Verified, beating Llama 4 Scout on reasoning despite a quarter of the parameters, and runs roughly 50 to 70 tokens per second on an RTX 4090. ^[6][10] gpt-oss-20b, OpenAI's open MoE (21B total, 3.6B active, Apache 2.0), is the speed champion: it ships in native MXFP4 at about 13 GB, hits around 225 tokens per second on an RTX 4090 and 45 to 50 on an M4 MacBook, and scores 85.3 MMLU and 74.2 GPQA Diamond, roughly OpenAI o3-mini class. ^[4][10] Qwen3-Coder 30B-A3B (about 19 GB at Q4) is the best local coding agent, and Gemma 4 31B, the family's dense flagship at MMLU-Pro 85.2, also fits 24 GB at Q4 (~18 GB) if you want the highest-quality multimodal option. ^[1][3]

Best local LLM for 48 GB and up

Once you have 48 GB or more (a 64 to 128 GB Apple Silicon Mac, a multi-GPU rig, or an 80 GB data-center card), you can run the largest openly available models.

gpt-oss-120b (117B total, 5.1B active, Apache 2.0) is the strongest model that fits a single accelerator. In MXFP4 the weights are about 63 GB and run on one 80 GB GPU, scoring 90.0 on MMLU and 80.9 on GPQA Diamond, near parity with OpenAI o4-mini on core reasoning, at roughly 315 tokens per second on a single 80 GB card. ^[4][13] Llama 4 Scout is the big-context multimodal option: 17B active drawn from a 109B, 16-expert MoE pool with an advertised context of up to 10M tokens. At Q4_K_M it needs about 60 GB (so 64 GB and up), while Unsloth's dynamic quant compresses it to roughly 32 GB to fit 48 GB with quality tradeoffs. ^[7][8] Note that Scout is the last open-weight Llama: Meta ended the brand and went closed in April 2026, so the model is stable but no longer maintained. ^[7] If you have exactly 48 GB and want the highest quality that fits cleanly, run Gemma 4 31B unquantized or at Q8, or Qwen3.6-35B-A3B (35B total, 3B active, about 24 GB at Q4, SWE-bench Verified 73.4), which stays very fast because of its sparse MoE design. ^[1][6]

How do I run these models: Ollama, LM Studio, llama.cpp, and MLX

Four tools cover almost every setup, and they share the same GGUF model format (except MLX).

Ollama is the fastest way to start. It is a command-line tool built on top of llama.cpp that pulls and runs models like container images: ollama run gemma4:12b downloads the weights, picks a sensible quant, and starts an OpenAI-compatible API at localhost:11434. Since v0.19 (March 2026) it also has an experimental MLX backend for Apple Silicon Macs with 32 GB or more. ^[3][11]

LM Studio is the best choice if you want a graphical app rather than a terminal. It has a built-in model browser, one-click chat, and a local server, and it can load both GGUF (via llama.cpp) and MLX builds, choosing the faster one on a Mac. ^[11]

llama.cpp is the C and C++ engine underneath most of the ecosystem. It runs GGUF models on Apple Silicon, CUDA, ROCm, Vulkan, and plain CPU, and offers the widest hardware and model coverage. Use it directly when you want maximum control or an unusual platform. ^[11]

MLX is Apple's framework built for the unified memory of M-series chips. Running MLX-format weights through mlx-lm typically delivers 20 to 40 percent higher generation throughput and about 10 percent lower memory use than the same quant on llama.cpp, so on a Mac prefer an MLX build when one exists. ^[11] For fine-tuning, Unsloth and vLLM are the common next steps, and GPT4All is another friendly desktop option.

How to pick a quantization

Quantization shrinks weights from 16-bit to about 4 bits so a model fits consumer memory. Q4_K_M is the default sweet spot: near-full quality at roughly a quarter of the size. Move up to Q5_K_M or Q6_K if you have spare memory and want a little more accuracy, or down to dynamic 2 to 3 bit quants only to make a large model fit at all. The gpt-oss models are an exception: they ship in native MXFP4 from day one, so there is no separate Q4 conversion. Total memory needed is the weight size plus the KV cache, which scales with context length, so a long context can add several gigabytes on top of the numbers above. ^[4][8][14]

References

1. Google, "Gemma 4 model card," Google AI for Developers, ai.google.dev/gemma/docs/core/model_card_4 (2026). ↩
2. Google, "Gemma 4: byte for byte, the most capable open models," The Keyword / Google blog, April 2, 2026. ↩
3. Ollama model library (gemma4, qwen3.6, gpt-oss, qwen3-coder), ollama.com/library, accessed July 2026. ↩
4. OpenAI, "gpt-oss-120b and gpt-oss-20b model card," openai.com and arXiv:2508.10925 (2025). ↩
5. Microsoft, "Phi-4-mini Technical Report," arXiv:2503.01743 (2025). ↩
6. Qwen Team (Alibaba), "Qwen3.6-27B: flagship-level coding in a 27B dense model," qwen.ai/blog, April 2026. ↩
7. Meta, "The Llama 4 herd," ai.meta.com, April 5, 2025. ↩
8. Unsloth, "Llama 4 and gpt-oss: how to run and fine-tune," unsloth.ai/docs (2026). ↩
9. Hugging Face, bartowski and unsloth "DeepSeek-R1-Distill-Qwen-14B GGUF" model cards, huggingface.co (2025-2026). ↩
10. runaihome, "gpt-oss-20b" and "Qwen3.6-27B local AI hardware guides" (tokens/sec figures), 2026. ↩
11. "A Comparative Study of MLX, MLC-LLM, Ollama, and llama.cpp," arXiv:2511.05502, and Contra Collective, "llama.cpp vs MLX vs Ollama on Apple Silicon," 2026. ↩
12. APXML, "Phi-4-mini: specifications and GPU VRAM requirements," apxml.com/models/phi-4-mini (2026). ↩
13. Baseten, "How we run gpt-oss-120b at 500+ tokens per second on NVIDIA GPUs," baseten.co, 2026. ↩
14. Artificial Analysis, open-weights model comparison and quantization notes, artificialanalysis.ai, accessed July 2026. ↩