AI Wiki

On-device AI

AI HardwareAI InferenceMachine LearningSmall Language Models
18 min read
Updated
Suggest editHistoryTalk50 citationsFact-checked
RawGraph

On-device AI is the practice of running machine learning models on the phone, laptop, watch, or embedded board a person is actually using, instead of sending the input to a remote data center. The category spans wake-word detectors, camera pipelines, translation models, and, since roughly 2023, small language models that fit inside a few gigabytes of memory. It overlaps heavily with edge AI and edge computing, which also cover gateways, cameras, and factory equipment that are not personal devices.

The arguments for local execution are consistent across vendors. Microsoft's documentation for Windows ML lists three: models run on the user's hardware, which keeps data private, removes cloud inference costs, and works without an internet connection [1]. Google's Android documentation for Gemini Nano makes the same three points and adds that executing prompts locally removes network latency, while cautioning that inference speed itself depends on device hardware; it also notes that the on-device stack runs inside AICore, a system service that follows Android's Private Compute Core rules, isolating each request and storing no record of inputs or outputs [2]. Apple frames Apple Intelligence the same way, saying that "many of the models that power it run entirely on device" [3].

In practice most shipping products are hybrid. Apple pairs local models with Private Cloud Compute for heavier requests [3], and Microsoft's Windows team describes the arrangement plainly: "The future of AI is hybrid, utilizing the respective strengths of cloud and client while harnessing every Windows device to achieve more" [33]. The practical question is therefore which parts of a workload fit inside a device's memory, power, and thermal budget.

What makes local inference hard

The binding constraint is usually memory, not arithmetic. The MobileLLM authors argued that a mobile app "should not exceed 10% of the DRAM, since DRAM is shared with the operating system and other applications," and calculated that a 7B-parameter model consumes about 0.7 joules per token; a fully charged iPhone holds roughly 50 kJ, so at 10 tokens per second such a model would drain the battery in under two hours. A 350M model at 8 bits, consuming 0.035 joules per token, would instead support conversational use for a full day [5].

Storage bandwidth matters as much as capacity. Apple's "LLM in a flash" work keeps weights in NAND flash and streams them in on demand, using windowing to reuse recently active neurons and row-column bundling to match flash's preference for large sequential reads. The authors report running models up to twice the size of available DRAM, with 4-5x faster inference than naive loading on CPU and 20-25x on GPU [6].

Disk footprint is a real product constraint too. Chrome's built-in Summarizer API, available from Chrome 138 stable, requires at least 22 GB of free space on the volume holding the user's profile, plus either more than 4 GB of VRAM or a CPU with 16 GB of RAM and four or more cores [7]. On Windows, the GPU build of Phi Silica is not preinstalled and downloads several gigabytes through Windows Update on first use, which is why Microsoft tells developers to show a consent dialog first [44].

Energy budgets shape model choice at the small end. Google reported that an INT4 quantization-aware-trained version of Gemma 3 270M used 0.75% of a Pixel 9 Pro's battery across 25 conversations [8].

Making models small enough

Most on-device deployments combine four families of technique: quantization, pruning, distillation, and architecture design. Most of them predate the current generation of language models.

TechniqueRepresentative resultSource
Pruning plus trained quantization plus Huffman codingAlexNet reduced 35x, from 240 MB to 6.9 MB; VGG-16 reduced 49x, from 552 MB to 11.3 MB, with no loss of accuracyHan, Mao and Dally, 2015 [9]
Knowledge distillationAn ensemble's knowledge compressed into a single deployable model, improving a heavily used commercial acoustic modelHinton, Vinyals and Dean, 2015 [10]
Post-training weight quantization (GPTQ)175B-parameter models quantized to 3 or 4 bits in about four GPU hours, enabling single-GPU generative inferenceFrantar et al., 2022 [11]
Activation-aware weight quantization (AWQ)4-bit on-device LLMs and VLMs; TinyChat reports more than 3x speedup over FP16 on desktop and mobile GPUsLin et al., 2023 [12]
Efficient architecture designDepthwise separable convolutions with two global hyperparameters trading latency against accuracyHoward et al., MobileNets, 2017 [16]

Model compression in shipping products usually stacks several of these. Meta built Llama 3.2 1B and 3B by pruning the Llama 3.1 8B model and applying knowledge distillation from the logits of Llama 3.1 8B and 70B, then released quantized versions a month later using two schemes: QLoRA, which is quantization-aware training with LoRA adaptors, and SpinQuant, a post-training method. Averaged over the released models, the quantized builds were 56% smaller, used 41% less memory, and ran 2-4x faster, benchmarked on an Android OnePlus 12 [14][15].

Apple's on-device model went further on bit width. The 2024 version used a mixed 2-bit and 4-bit palettization scheme averaging 3.7 bits per weight [41]. The 2025 revision moved the decoder weights to 2 bits per weight using quantization-aware training, with 4 bits for embeddings and an 8-bit KV cache; it also splits the model into two blocks at a 5:3 depth ratio and shares the second block's key-value caches with the last layer of the first, cutting KV cache memory by 37.5% [42].

Sparsity is the newest lever. Apple's third-generation on-device model, AFM 3 Core Advanced, is a 20-billion-parameter sparsely activated mixture of experts network, built on a technique Apple calls Instruction-Following Pruning, that activates just 1 to 4 billion parameters at a time depending on the request. The full model sits in NAND flash. Because, in Apple's words, "NAND-to-DRAM bandwidth is too slow to swap weights token by token, as standard MoE models require," AFM 3 Core Advanced instead makes routing decisions per prompt: a lightweight dense block selects a fixed set of experts during initial processing and periodically reselects them during generation. A high percentage of the network is always-active shared experts, with input-dependent routed experts swapped into DRAM only when needed [19]. Google's Gemma 3n attacks the same problem differently: Per-Layer Embeddings move a large share of parameters to CPU-side computation so that only about 2B (E2B) or 4B (E4B) core transformer weights need accelerator memory, and its MatFormer design nests a smaller working model inside the larger one [17][18].

Speculative decoding is common at the runtime layer. Microsoft uses a small draft model to propose token sequences that the main model validates in parallel, a path available on NPU-equipped Windows devices but not on the GPU build [44].

Silicon

Dedicated neural accelerators arrived in consumer phones in 2017. Apple's A11 Bionic included a dual-core neural engine that Apple said performs up to 600 billion operations per second, used for Face ID and Animoji [20]. Nine years later, the class of hardware described as an NPU is standard across phones, laptops, and even watches.

PlatformAnnouncedStated AI hardware
Apple A11 BionicSeptember 2017Dual-core neural engine, up to 600 billion operations per second [20]
Google Coral Edge TPU-4 TOPS at 2 TOPS per watt, integer-quantized TensorFlow Lite models only [27]
Copilot+ PC baselineMay 2024NPU of at least 40 TOPS; Snapdragon X Series rated 45 NPU TOPS [24]
Qualcomm Snapdragon 8 Elite Gen 5September 2025Hexagon NPU "37% faster performance" [22]
Apple M5October 2025Neural Accelerator in each of 10 GPU cores, over 4x peak GPU compute for AI versus M4, 16-core Neural Engine, 153 GB/s memory bandwidth [21]
MediaTek Dimensity 9500December 2025 (LiteRT support)Gemma models up to 12x faster than CPU and 10x faster than GPU on the NPU [28]
Qualcomm Snapdragon Wear EliteMarch 2026Hexagon NPU supporting "up to billion-parameter models at the edge" [23]

Microsoft defines the Copilot+ PC class by hardware floor rather than by chip vendor: an NPU capable of at least 40 TOPS, 16 GB of RAM, a 256 GB SSD, and Windows 11 version 24H2 or newer [25]. The first wave used Qualcomm Snapdragon X Elite and X Plus parts and went on sale on 18 June 2024 from $999 [24]; Intel Core Ultra 200V and AMD Ryzen AI 300 series machines followed [26].

NPUs impose their own conversion tax. Microsoft's developer guidance notes that models are typically trained in FP32 while "many NPU devices, however, only support integer math in lower bit format, such as INT8," so models must be quantized and compiled before they will run on the accelerator [26].

Runtimes and frameworks

The tooling layer is crowded, partly because each silicon vendor exposes its accelerator differently.

RuntimeOriginScope
Core ML and coremltoolsAppleUses CPU, GPU, and Neural Engine; coremltools converts TensorFlow, PyTorch, scikit-learn, XGBoost and LIBSVM models [29]
LiteRT (formerly TensorFlow Lite)GoogleRenamed on 4 September 2024; runs models authored in PyTorch, JAX, and Keras [30]. Part of Google AI Edge alongside LiteRT-LM, MediaPipe, and Model Explorer [31]
ONNX RuntimeMicrosoftRuns on Linux, Windows, macOS, iOS, Android, and browsers, across CPU, GPU, and NPU [32]
Windows MLMicrosoftGenerally available 23 September 2025; ONNX Runtime under the hood with execution providers from AMD (Vitis AI), Intel (OpenVINO), NVIDIA (TensorRT for RTX), and Qualcomm (QNN), delivered through Windows Update [33][1]
ExecuTorchMeta and the PyTorch projectReached beta at v0.4 in October 2024; backends include XNNPACK, Core ML, Vulkan, Qualcomm, MediaTek, Arm Ethos-U, and Cadence DSP [34][35]
llama.cppGeorgi Gerganov and contributors"LLM inference in C/C++" with 1.5-bit to 8-bit integer quantization and backends including Metal, CUDA, Vulkan, OpenCL, and Hexagon [36]
MLC LLMMLC AI"A machine learning compiler and high-performance deployment engine for large language models," targeting Metal on Apple A-series GPUs, OpenCL on Adreno and Mali GPUs, and WebGPU in the browser [37]
MLXApple machine learning research"An array framework for machine learning on Apple silicon" with a unified memory model, so arrays are shared between CPU and GPU without transfers [38]

The GGUF container, created by the author of llama.cpp, is the de facto distribution format for locally run open-weight models. It stores tensors together with standardized metadata and defines a large set of quantization types, from 8-bit legacy blocks down to schemes such as Q4_K at 4.5 bits per weight and IQ1_S at 1.56 bits per weight [13]. Tools including Ollama, LM Studio, and GPT4All consume it [13].

On-device language models

ModelVendorDateSizeNotes
Gemini NanoGoogleDecember 2023Not disclosedShips inside AICore on Android 14, first on Pixel 8 Pro [39]
Phi-3-miniMicrosoftApril 20243.8B, 3.3T training tokens69% MMLU, 8.38 MT-bench; at 4 bits it occupies about 1.8 GB and ran at more than 12 tokens per second on an iPhone 14 [43]
Apple on-device foundation modelAppleJune 2024About 3B3.7 bits per weight; time to first token of about 0.6 ms per prompt token and generation at 30 tokens per second on iPhone 15 Pro [41]
Llama 3.2 1B and 3BMeta AISeptember 20241B and 3B, 128K contextPruned and distilled from Llama 3.1; built with Qualcomm and MediaTek hardware support and optimized for Arm processors [15]
Apple on-device model, 2025 revisionAppleJune 2025About 3B2 bits per weight via QAT; exposed to apps through the Foundation Models framework [42]
Gemma 3n E2B and E4BGoogleJune 20255B and 8B raw parametersRun in roughly 2 GB and 3 GB; E4B passed 1300 on LMArena, which Google called the first sub-10B model to do so [17]
Gemma 3 270MGoogleAugust 2025270M (170M embedding, 100M transformer)INT4 QAT checkpoints; 0.75% of a Pixel 9 Pro battery for 25 conversations [8]
Gemma 4 E2B and E4B, Gemini Nano 4GoogleApril 2026 (AICore developer preview)E2B and E4BGemma 4 is the base model for Gemini Nano 4, reported as up to 4x faster than the previous version using up to 60% less battery [40]
AFM 3 Core and AFM 3 Core AdvancedAppleJune 20263B dense; 20B sparse with 1-4B activeCore Advanced is stored in flash with experts loaded into DRAM on demand [19]

Google says Gemini Nano now runs on more than 140 million devices and that developers reach it through ML Kit's GenAI APIs, which cover prompting, summarization, proofreading, rewriting, image description, and speech recognition [4][2]. Apple's Foundation Models framework, announced on 9 June 2025 at WWDC, gives Swift apps direct access to the local model with guided generation through a @Generable macro [42].

Apple is unusually explicit about what these models are for. Its 2025 write-up states that the on-device model "is not designed to be a chatbot for general world knowledge," and lists summarization, entity extraction, text understanding, refinement, short dialog, and creative content generation as the intended uses [42].

Platform and browser integration

Browsers have become a distribution channel for local models. Chrome exposes Prompt, Summarizer, Translator, Language Detector, Proofreader, Writer, and Rewriter APIs, built on foundation and expert models shipped with the browser; the Prompt API runs on Gemini Nano [46]. Microsoft Edge announced on 2 June 2026 that Translator and Language Detector APIs, backed by task-specific models built into the browser and supporting more than 145 languages, were available in Edge 148, and that its Canary and Dev channels had gained on-device speech recognition through the Web Speech API plus a pre-release local model, Aion-1.0-Instruct. Microsoft describes Aion-1.0-Instruct as smaller, faster, and more efficient than the model it succeeds, Phi-4-mini (which the post calls a 4B-parameter model), extending support to devices with less capable GPUs and, through CPU inference, to devices with no GPU at all [45].

On Windows, Phi Silica is the built-in local language model for Copilot+ PCs, reachable through the Windows App SDK. It has since been extended to non-Copilot+ machines with NVIDIA GeForce RTX 30 series or newer and AMD Radeon RX 9060 series or newer GPUs with 6 GB or more of VRAM, though that path currently requires Developer Mode and an Experimental Channel Windows Insider build. Microsoft has published a migration timeline off Phi Silica and onto Aion Instruct: a standalone sideloadable package in September 2026, Windows Insider rollout in October 2026, and retail rollout with Phi Silica removed in November 2026 [44].

Windows Recall, the feature that periodically snapshots the screen and indexes it locally, began rolling out in phases to Copilot+ PCs on 25 April 2025. Microsoft describes it as opt-in, requiring Windows Hello biometric enrollment, with snapshots encrypted and "processed in an isolated memory environment, a protected area in a computer's memory that processes sensitive information and prevents unauthorized access" [47]. Microsoft's user documentation still calls it Recall (preview) and still restricts it to Copilot+ PCs meeting the Secured-core standard [50]. It is the clearest example that keeping processing local does not by itself settle privacy questions.

Limits

Small models fail differently from large ones, and the failures reach users. Apple disabled Apple Intelligence notification summaries for news and entertainment apps in the iOS 18.3, iPadOS 18.3, and macOS Sequoia 15.3 developer betas in January 2025 after the BBC reported that a summary had falsely claimed Luigi Mangione, who had been charged in a murder case, "had shot himself"; the surviving summaries were italicized and labeled as beta [48]. In March 2025 Apple confirmed that the more personalized Siri, the version able to use personal context and act inside apps, would slip, saying it would "take us longer than we thought to deliver" [49].

Fragmentation is the other persistent cost. A model that runs on an Apple Neural Engine needs a different conversion path than one targeting Hexagon, an Intel NPU, or an Arm Ethos-U, and each vendor ships its own execution provider or delegate. Windows ML's design, downloading vendor execution providers through Windows Update rather than bundling them in each app, is a direct response to that problem [1]. Availability is also uneven for reasons that have nothing to do with silicon: Microsoft notes that Phi Silica features "are not available in China" [44].

Finally, the headline hardware numbers are hard to compare. TOPS ratings are quoted at different precisions and by different vendors, and reported throughput depends heavily on context length and quantization. Google's figures for MediaTek hardware illustrate the spread: Gemma 3n E2B on a Dimensity 9500 NPU reaches "over 1600 tokens/sec for prefill" but "28 tokens/sec for decode (with 4K context)" [28].

See also

References

  1. ^Microsoft Learn, "What is Windows ML?" learn.microsoft.com/...overview
  2. ^Android Developers, "Gemini Nano." developer.android.com/...gemini-nano
  3. ^Apple Newsroom, "Apple Intelligence is available today on iPhone, iPad, and Mac," 28 October 2024. apple.com/...vailable-today-on-iphone-ipad-and-mac
  4. ^Android Developers Blog, "Build intelligent Android apps: on-device inference," 21 July 2026. android-developers.googleblog.com/...ice-inference
  5. ^Z. Liu et al., "MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases," arXiv:2402.14905, February 2024. arxiv.org/...2402.14905
  6. ^K. Alizadeh et al., "LLM in a flash: Efficient Large Language Model Inference with Limited Memory," arXiv:2312.11514, December 2023. arxiv.org/...2312.11514
  7. ^Chrome for Developers, "Summarizer API." developer.chrome.com/...summarizer-api
  8. ^Google Developers Blog, "Introducing Gemma 3 270M: the compact model for hyper-efficient AI," 14 August 2025. developers.googleblog.com/...roducing-gemma-3-270m
  9. ^S. Han, H. Mao and W. J. Dally, "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding," arXiv:1510.00149, October 2015. arxiv.org/...1510.00149
  10. ^G. Hinton, O. Vinyals and J. Dean, "Distilling the Knowledge in a Neural Network," arXiv:1503.02531, March 2015. arxiv.org/...1503.02531
  11. ^E. Frantar, S. Ashkboos, T. Hoefler and D. Alistarh, "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers," arXiv:2210.17323, October 2022. arxiv.org/...2210.17323
  12. ^J. Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration," arXiv:2306.00978, June 2023. arxiv.org/...2306.00978
  13. ^Hugging Face Hub documentation, "GGUF." huggingface.co/...gguf
  14. ^Meta AI, "Introducing quantized Llama models with increased speed and a reduced memory footprint," 24 October 2024. ai.meta.com/...-llama-quantized-lightweight-models
  15. ^Meta AI, "Llama 3.2: Revolutionizing edge AI and vision with open, customizable models," 25 September 2024. ai.meta.com/...ect-2024-vision-edge-mobile-devices
  16. ^A. G. Howard et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications," arXiv:1704.04861, April 2017. arxiv.org/...1704.04861
  17. ^Google Developers Blog, "Introducing Gemma 3n: the developer guide," 26 June 2025. developers.googleblog.com/...ma-3n-developer-guide
  18. ^Google AI for Developers, "Gemma 3n model overview." ai.google.dev/...gemma-3n
  19. ^Apple Machine Learning Research, "Introducing the Third Generation of Apple's Foundation Models," 8 June 2026. machinelearning.apple.com/...ple-foundation-models
  20. ^Apple Newsroom, "The future is here: iPhone X," 12 September 2017. apple.com/...the-future-is-here-iphone-x
  21. ^Apple Newsroom, "Apple unleashes M5, the next big leap in AI performance for Apple silicon," 15 October 2025. apple.com/...p-in-ai-performance-for-apple-silicon
  22. ^Qualcomm, "Snapdragon 8 Elite Gen 5, the World's Fastest Mobile System on a Chip," 24 September 2025. qualcomm.com/...world-s-fastest-mobile-system-on-a
  23. ^Qualcomm, "Qualcomm Powers the Rise of Personal AI with New Snapdragon Wear Elite Platform," 2 March 2026. qualcomm.com/...rsonal-ai-with-new-snapdragon-wear
  24. ^Microsoft, "Introducing Copilot+ PCs," 20 May 2024. blogs.microsoft.com/...introducing-copilot-pcs
  25. ^Microsoft Windows Learning Center, "Copilot+ PCs and Windows PCs: what's the difference?" microsoft.com/...-plus-pcs-windows-pcs-differences
  26. ^Microsoft Learn, "Copilot+ PCs developer guide." learn.microsoft.com/...npu-devices
  27. ^Coral, "Edge TPU FAQ." coral.ai/...faq
  28. ^Google Developers Blog, "MediaTek NPU and LiteRT: powering the next generation of on-device AI," 8 December 2025. developers.googleblog.com/...ation-of-on-device-ai
  29. ^Apple, coremltools documentation, "Overview." apple.github.io/...overview-coremltools
  30. ^Google Developers Blog, "TensorFlow Lite is now LiteRT," 4 September 2024. developers.googleblog.com/...ow-lite-is-now-litert
  31. ^Google AI Edge. developers.google.com/edge
  32. ^ONNX Runtime. onnxruntime.ai
  33. ^Windows Developer Blog, "Windows ML is generally available: empowering developers to scale local AI across Windows devices," 23 September 2025. blogs.windows.com/...cal-ai-across-windows-devices
  34. ^PyTorch Blog, "ExecuTorch Beta: on-device AI and LLMs, stability, and acceleration with partners," 24 October 2024. pytorch.org/...executorch-beta
  35. ^ExecuTorch repository, PyTorch. github.com/...executorch
  36. ^llama.cpp repository, ggml-org. github.com/...llama.cpp
  37. ^MLC LLM repository, MLC AI. github.com/...mlc-llm
  38. ^MLX repository, Apple machine learning research. github.com/...mlx
  39. ^Android Developers Blog, "A new foundation for AI on Android," 6 December 2023. android-developers.googleblog.com/...ai-on-android
  40. ^Android Developers Blog, "Gemma 4: the new standard for local agentic intelligence on Android," 2 April 2026. android-developers.googleblog.com/...-intelligence
  41. ^Apple Machine Learning Research, "Introducing Apple's On-Device and Server Foundation Models," 10 June 2024. machinelearning.apple.com/...ple-foundation-models
  42. ^Apple Machine Learning Research, "Updates to Apple's On-Device and Server Foundation Language Models," 9 June 2025. machinelearning.apple.com/...n-models-2025-updates
  43. ^M. Abdin et al., "Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone," arXiv:2404.14219, April 2024. arxiv.org/...2404.14219
  44. ^Microsoft Learn, "Get started with Phi Silica in the Windows App SDK." learn.microsoft.com/...phi-silica
  45. ^Microsoft Edge Blog, "Expanding on-device AI in Microsoft Edge: new models and APIs for the web," 2 June 2026. blogs.windows.com/...w-models-and-apis-for-the-web
  46. ^Chrome for Developers, "Built-in AI." developer.chrome.com/...built-in
  47. ^Windows Experience Blog, "Responsible AI and trustworthy innovation on Windows," 25 April 2025. blogs.windows.com/...tworthy-innovation-on-windows
  48. ^TechCrunch, "Apple pauses AI notification summaries for news after generating false alerts," 16 January 2025. techcrunch.com/...ws-after-generating-false-alerts
  49. ^TechCrunch, "AI-powered 'more personalized Siri' is delayed," 7 March 2025. techcrunch.com/...ore-personalized-siri-is-delayed
  50. ^Microsoft Support, "Retrace your steps with Recall." support.microsoft.com/...8b-4b3e-b0a1-2eb8ac48701c

Improve this article

Add missing citations, update stale details, or suggest a clearer explanation. Every suggestion is reviewed for sourcing before it goes live.

v1 · 3,643 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent adversarial fact-check at creation (wanted175 campaign, 2026-07-24): every claim verified against primary sources by a dedicated verification agent; corrections applied before publication.

Cite this page: AI Wiki. "On-device AI." aiwiki.ai, updated 24 Jul 2026, fact-checked 24 Jul 2026. CC BY 4.0. https://aiwiki.ai/wiki/on_device_ai

Suggest edit