MLC LLM
MLC LLM (Machine Learning Compilation for Large Language Models, also written MLC-LLM) is an open-source compiler and runtime for running large language models on a wide range of hardware.[8] Its documentation calls it "a machine learning compiler and high-performance deployment engine for large language models".[1] Rather than relying on hand-written GPU kernels for every platform, it uses the Apache TVM compiler to generate a binary library for a given model and device, and runs that library through a single runtime, MLCEngine, which offers an OpenAI-compatible API from Python, a REST server, JavaScript, Swift and Kotlin.[1][2][6]
The code was first published in April 2023 by members of Carnegie Mellon University's Catalyst group, the University of Washington's SAMPL lab, Shanghai Jiao Tong University, the company OctoML and the wider MLC community.[4][5] Supported targets, according to the project's README, are AMD, NVIDIA, Apple and Intel GPUs on Linux, Windows and macOS (through Vulkan, CUDA, ROCm and Metal), iPhones and iPads, Android phones with Adreno or Mali GPUs, and web browsers through WebGPU and WebAssembly.[2] The browser target is maintained as a separate sibling project, WebLLM.[10][17]
MLC LLM itself is licensed under the Apache License 2.0.[3][4] The models it runs are open-weight models published by other organizations, each under its own terms.[46] On 19 November 2026 the GitHub repository had 23,382 stars and 2,190 forks, and the most recent commit on its main branch was dated 2026-11-16.[4][23] The only entry on the repository's Releases page at that date was a pre-release, v0.1.dev0, published on 2023-04-29. The most recently dated version tag, v0.20.0, is an annotated tag of 2026-07-07; another tag, v0.26.dev0, points to a main-branch commit dated 2026-06-28.[22][58]
History
Origins in 2023
The mlc-ai organization on GitHub published two browser projects shortly before MLC LLM. Its Web Stable Diffusion repository was created on 2023-03-06 and its WebLLM repository on 2023-04-13; the MLC LLM repository followed on 2023-04-29.[4][9][10] Its initial commit was made from the GitHub account spectrometerHBH, whose profile name is Bohan Hou, and lists Hongyi Jin, Ruihang Lai, Tianqi Chen, Junru Shao and Zihao Ye as co-authors.[11]
The project was introduced in a blog post of May 1, 2023, signed "MLC Community". The post argued that language models were likely to migrate to consumer devices such as laptops, phones and game consoles, and that one obstacle was the variety of consumer hardware and GPU programming interfaces, for which it listed CUDA, ROCm, Metal, OpenCL, SYCL, Direct3D, Vulkan/SPIR-V and WebGPU. Machine learning compilation was presented as the answer: treat the model as a program and transform it step by step into code for each target. The first version mapped models to Vulkan and Metal, which the post said covered the majority of consumer platforms, including Windows, Linux and macOS, and came with a command-line demo and an iPhone demo app distributed through TestFlight. The post said MLC LLM was built on the Apache TVM community's TVM Unity effort.[5]
A week later, on May 8, 2023, the project announced Android support. The post described running a Vicuna 7B model on a Samsung Galaxy S23 by generating OpenCL kernels for the phone's Adreno GPU, and noted that the team used a 4-bit quantized model on Android because Android lacked the 4 GB per-app memory limit that iOS enforced.[12] On May 22, 2023 a third post covered the RedPajama 3B model on Apple silicon, AMD and NVIDIA GPUs, iPhones and browsers, in a collaboration that the post attributes to ETH Zürich, Together, OctoML, CMU Catalyst and the MLC community.[13]
AMD and multi-GPU work
In August 2023 the project published results for AMD graphics cards using ROCm, and showed a 4-bit quantized 7B Llama model running on a Steam Deck through Vulkan.[14] In October 2023 it added multi-GPU inference, modeled in TVM Unity as a single-program-multiple-data representation with tensor parallelism, and reported results for Llama 2 70B and Code Llama 34B on pairs of consumer GPUs and on up to eight server GPUs.[15] The code samples in these posts show the Python interface of the time: a package named mlc_chat with a ChatModule class.[14][15] A post dated April 20, 2024 described running models on the Mali GPU of an Orange Pi 5 single-board computer through TVM's OpenCL backend.[16]
MLCEngine (2024)
On June 7, 2024 the project introduced the MLC LLM Engine, or MLCEngine. The announcement said that earlier versions of MLC LLM, like many inference projects, had used separate implementations for server and local use, and that MLCEngine replaced them with one engine for both. It combined server techniques such as continuous batching, speculative decoding, paged KV cache management and prefix caching with the project's existing portability, and exposed the same OpenAI-style chat completion call in Python, Swift, Kotlin and JavaScript and through a REST server. The post called this "a new chapter of the MLC LLM project" and credited contributions from CMU Catalyst, OctoAI, UW SAMPL, SJTU and the broader MLC community.[6]
Six days later, on June 13, 2024, the WebLLM engine was presented as the web backend of MLCEngine.[17] In October 2024 the project published a study of throughput and latency on NVIDIA H100 GPUs, and in November 2024 it released XGrammar, a library for structured output that MLC LLM uses through its C++ core.[18][19]
2025 and 2026
In January 2025 the project described MicroServing, a set of three sub-request calls (prep_recv, remote_send and start_generate) that let a programmable router combine several engines, for example to run prompt processing and token generation on different engines (prefill-decode disaggregation).[20] The documentation includes a tutorial for writing such a router with mlc_llm.router.[56]
A May 2026 post on XGrammar-2 listed MLC LLM among the engines that had integrated the new version, alongside SGLang, vLLM and TensorRT-LLM.[21] Commits to the main branch in October and November 2026 included fixes for Gemma 3 attention, updates of the bundled TVM, and new model definitions for LFM2 and SmolLM3.[23]
| Date | Event |
|---|---|
| 2023-04-29 | Repository created; pre-release v0.1.dev0 published [4][58] |
| 2023-05-01 | Introductory blog post; Vulkan and Metal targets, iPhone app [5] |
| 2023-05-08 | Android support through OpenCL [12] |
| 2023-05-22 | RedPajama support, including WebGPU and iOS [13] |
| 2023-08-09 | AMD ROCm results published [14] |
| 2023-10-19 | Multi-GPU inference [15] |
| 2024-04-20 | Orange Pi (Mali GPU) post [16] |
| 2024-06-07 | MLCEngine announced [6] |
| 2024-06-13 | WebLLM engine announced [17] |
| 2024-10-10 | H100 throughput and latency study [18] |
| 2024-11-22 | XGrammar released [19] |
| 2025-01-07 | MicroServing described [20] |
| 2026-07-07 | Version tag v0.20.0 created [22] |
| 2026-11-16 | Latest commit on main as of 19 November 2026 [23] |
Compilation approach
Machine learning compilation
The project uses "machine learning compilation" (MLC) to mean compiling and automatically optimizing a machine learning workload for a target backend, in place of writing kernels by hand for each one.[14] In the flow the project described in 2023, a model's execution logic is captured in a TVM container called an IRModule, which holds functions for steps such as prompt encoding and single-token decoding. The compiler then transforms that module: it can replace operators such as attention with library or generated implementations, plan memory so that layers that never run at the same time share buffers, fuse operators, and finally generate kernel code for the target, for example WGSL for the web and SPIR-V for native devices from the same pipeline. The project presents the approach as complementary to hand optimization: vendor libraries are used where they exist and generated code covers the rest.[5]
The term is broader than this one project. The MLC community's site defines a machine learning compiler as "a specialized compiler that transforms high-level ML models into optimized code that can efficiently run on various hardware platforms".[54] The TVM paper of 2018 proposed "a compiler that exposes graph-level and operator-level optimizations to provide performance portability to deep learning workloads across diverse hardware back-ends".[24] The introductory post named PyTorch 2.0, OpenXLA/MLIR and Apache TVM Unity as open-source developments in machine learning compilation, and the MLC community also publishes a course on the general subject.[5]
TVM, Relax and TensorIR
MLC LLM is built on Apache TVM, a compiler described in a 2018 paper by Tianqi Chen and eleven co-authors, presented at the USENIX OSDI symposium.[24] It depends specifically on "TVM Unity", which the documentation describes as the latest development in Apache TVM and as required to build MLC LLM.[27] The repository takes its TVM source from the mlc-ai/relax repository, included as a Git submodule.[28]
Two abstractions from that line of work are described in separate papers. Relax, described by Ruihang Lai and colleagues in a paper published at ASPLOS 2025, represents computational graphs, loop-level tensor programs and external library calls in one program, and tracks symbolic tensor shapes across the whole program. Its authors describe such dynamic-shape computation as critical for large language models.[25] TensorIR, published at ASPLOS 2023, generalizes the loop-nest representation of earlier machine learning compilers.[26] The project's AMD and Mali ports both describe reusing a generic GPU kernel optimization space written in TensorIR and retargeting it to the new device.[14][16]
Model architectures are written in Python with TVM's tvm.nn.module interface, whose syntax the project describes as similar to PyTorch.[13][29] Adding a variant of an architecture that is already supported, such as a fine-tuned Llama or Mistral model, needs only a weight conversion; a new architecture needs a new model definition.[29] A paper by Siyuan Feng and colleagues, first posted in April 2024, describes TapML, the testing and debugging method used in the project, and states that it had been used to deploy 105 models in 27 architectures on five platforms within two years.[30]
Workflow and artifacts
Running a model needs two artifacts: model weights converted to MLC format, and a compiled model library that contains the inference logic.[31] The documented steps for a model that has not been converted are:
| Step | Command | Purpose |
|---|---|---|
| Generate configuration | mlc_llm gen_config | Writes mlc-chat-config.json from the chosen quantization and conversation template [1][57] |
| Convert weights | mlc_llm convert_weight | Converts and quantizes Hugging Face weights to MLC format [1] |
| Compile library | mlc_llm compile | Produces the model library for a device [1][31] |
| Package for mobile | mlc_llm package | Builds the model libraries and runtime for an iOS or Android app [37][38] |
The explicit compile step is optional for the Python API, the chat command and the REST server. When the engine starts without a library, it compiles one just in time and caches it, so a first run consists of downloading pre-quantized weights, compiling, and then starting the chat runtime; later runs reuse the cache.[1] Compiled libraries are .so files on Linux and Apple-silicon Macs, .dylib on Intel Macs, .dll on Windows, .wasm for the web, and .tar archives for iPhone and Android.[1][57] The same converted weights are shared by all platforms.[31] As of November 2026 the documentation also describes an opt-in model artifact manifest, a file named mlc-model-manifest.json that ties converted weights, a compiled model library and a frontend together; a model without one loads from mlc-chat-config.json as before.[62]
Quantization modes are named by a short code of the form qAfB(_id), where A is the number of bits used for weights and B the number used for activations. The documented options are q0f16, q0f32, q3f16_1, q4f16_1, q4f32_1 and q4f16_awq, the last of which the documentation marks as not stable. On CUDA the project also supports FP8 weight-and-activation quantization, which requires a calibration run.[32]
Platforms and backends
The README gives the following support matrix.[2]
| Platform | AMD GPU | NVIDIA GPU | Apple GPU | Intel GPU |
|---|---|---|---|---|
| Linux / Windows | Vulkan, ROCm | Vulkan, CUDA | n/a | Vulkan |
| macOS | Metal (discrete GPU) | n/a | Metal | Metal (integrated GPU) |
| Platform | Backend |
|---|---|
| Web browser | WebGPU and WebAssembly |
| iOS / iPadOS | Metal on Apple A-series GPUs |
| Android | OpenCL on Adreno and Mali GPUs |
The --device option of mlc_llm serve accepts cuda, metal, vulkan, rocm, opencl or auto.[7] As of November 2026 the installation page offers nightly Python wheels from the project's own index for CPU, CUDA 12.8, CUDA 13.0, ROCm 6.1, ROCm 6.2 and Vulkan on Linux, with Metal on macOS and Vulkan on Windows, and recommends a Conda environment with Python 3.13.[33] The CPU package is installed with:
python -m pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cpu mlc-ai-nightly-cpu
The 2024 MLCEngine announcement listed the NVIDIA GeForce RTX 4090, NVIDIA Jetson Orin, NVIDIA T4, AMD Radeon RX 7900 XTX, Steam Deck and Orange Pi among the devices on which the engine had been validated.[6]
MLCEngine and its interfaces
MLCEngine is the runtime that loads a compiled model library and serves requests. All of its language bindings use the same call shape, engine.chat.completions.create, modeled on OpenAI's chat completion API.[6]
| Interface | How it is exposed | Notes |
|---|---|---|
| Command line | mlc_llm chat | Interactive chat with commands such as /stats, /metrics and /reset [36] |
| REST server | mlc_llm serve | Listens on 127.0.0.1:8000 by default [7] |
| Python | mlc_llm.MLCEngine, mlc_llm.AsyncMLCEngine | The synchronous class does not batch concurrent requests [35] |
| iOS | MLCSwift Swift package | Streams results through Swift's AsyncStream [6][37] |
| Android | mlc4j Java library with a Kotlin API | Included in an app as a Gradle subproject [6][38] |
| Browser | WebLLM (@mlc-ai/web-llm) | Separate repository and npm package [39][40] |
REST server
The quick-start command for a server, using a 4-bit quantized Llama 3 8B model from the project's Hugging Face account, is:[1]
mlc_llm serve HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC
The REST reference documents two endpoints, GET /v1/models and POST /v1/chat/completions. The chat endpoint accepts the usual OpenAI request fields, including stream, temperature, top_p, logprobs, logit_bias, seed, tools, tool_choice and response_format.[7] The server's source code on the main branch also defines POST /v1/completions and POST /v1/embeddings routes, which that reference page does not describe.[7][60] The documentation's own wording is that the server has "complete OpenAI API support".[43] Function calling follows the OpenAI format, and response_format can request JSON output with an optional JSON schema.[7] An IDE integration guide describes using the REST API for code completion in an editor, with the editor extension pointed at the server's /v1/completions route.[57]
The server has three preset modes. local, the default, sets the maximum batch size to 4; interactive allows one request at a time; server infers the largest batch size and total sequence length that fit in GPU memory. Speculative decoding is selected with --speculative-mode, whose documented values are disable, small_draft, eagle and medusa, and it currently supports one additional draft model. Multi-GPU tensor parallelism is enabled with an override:[7]
mlc_llm serve HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC --overrides "tensor_parallel_shards=2"
Engine features
The June 2024 announcement listed continuous batching, speculative decoding, paged KV management, common prefix caching and cascade inference as the engine's main optimizations, and said that on CUDA it uses the FlashInfer library for attention kernels.[6] The October 2024 study added detail: matrix operations are dispatched to cuBLAS or CUTLASS kernels at high concurrency and to compiler-generated kernels at low concurrency, memory is planned statically by a compiler pass that understands dynamic shapes, CUDA Graph is used to cut kernel launch overhead, and the engine loop runs on its own thread, separate from tokenization and HTTP handling.[18]
Structured generation is built into the engine and therefore available on every platform. The June 2024 engine used a grammar state machine to constrain output to JSON or to a JSON schema. XGrammar, announced in November 2024 and included in the repository as a submodule, executes context-free grammars and is described in a paper by Yixin Dong and colleagues.[6][19][28][42]
Mobile apps
The project publishes a demonstration app, MLC Chat, for both mobile platforms.[43] The iOS version is distributed through Apple's App Store, where the listing names Tianqi Chen as the seller, says the app is intended for non-commercial purposes, and notes that "Each model can be subject to its respective licenses."[46] The Android version is distributed as an APK built for the Samsung S23 with a Snapdragon 8 Gen 2 chip.[38] Both apps download model weights from Hugging Face, or can bundle them at build time.[37][38]
WebLLM
WebLLM is the in-browser counterpart of MLC LLM. Its repository predates MLC LLM's by about two weeks, and its README calls it a companion project of MLC LLM.[10][40] Since June 2024 the project has described it as a specialized web backend of MLCEngine.[17][39]
MLC LLM compiles a model ahead of time into converted weights and a WebAssembly library. That library holds the compute kernels as WGSL shaders, which the browser's WebGPU implementation translates to native GPU code, together with non-kernel functions compiled to WebAssembly. The browser downloads both parts once and caches them. A web application talks to the engine through an OpenAI-style JavaScript API, and the heavy computation can be moved off the page's main thread into a web worker or service worker; a variant exists for Chrome extensions.[17] Compiling a model for the webgpu target requires building MLC LLM from source and setting up a WebAssembly build environment.[31]
A paper on WebLLM by Charlie F. Ruan and colleagues was posted to arXiv in December 2024 and revised in April 2026.[41] WebLLM is licensed under Apache 2.0. On 19 November 2026 its repository had 19,389 stars, and its latest release, v0.2.89, had been published two days earlier.[10]
Models
MLC LLM uses its own weight format and compiled libraries, where llama.cpp reads GGUF files.[1][47] Weights are converted from Hugging Face format, and the project hosts pre-converted weights under the mlc-ai account on Hugging Face, which listed 412 model repositories on 19 November 2026.[1][44] Converted models in the documentation are named with the model name, the quantization code and the suffix -MLC, as in Llama-3-8B-Instruct-q4f16_1-MLC.[1]
The 2024 announcement named Llama 3, Mistral and Mixtral, Phi-3, Qwen2 and Gemma among the supported families.[6] On 19 November 2026 the repository's model directory contained definitions named for further families, including Gemma 4, Llama 4, Qwen3, DeepSeek models, LLaVA, MiniCPM, Nemotron, RWKV, InternLM, OLMo and LFM2.[45]
These are open-weight models released by their own developers. MLC LLM's Apache 2.0 license covers the compiler and runtime only, and each model keeps the license of its weights.[3][46]
Published performance results
The figures below are the project's own, taken from its blog posts and papers. Each applies only to the hardware, model, quantization and software versions of its date.
| Date | Setup | Result reported by the project |
|---|---|---|
| 2023-08-09 | Llama 2 7B and 13B, 4-bit, single request, ROCm 5.6 | Radeon RX 7900 XTX reached 80% of the speed of an RTX 4090 and 94% of an RTX 3090 Ti; the post advises a 10% error bar [14] |
| 2023-10-19 | Llama 2 70B, 4-bit, two GPUs | 34.5 tokens/s on two RTX 4090; 29.9 tokens/s on two Radeon 7900 XTX [15] |
| 2023-10-19 | Code Llama 34B, 4-bit, two RTX 4090 | 64 tokens/s [15] |
| 2024-04-20 | Orange Pi 5, Mali-G610 GPU | 2.3 tokens/s for Llama 3 8B, 2.5 for Llama 2 7B, 5 for RedPajama 3B [16] |
| 2024-06-13 | WebGPU versus native Metal, M3 Max MacBook Pro, 4-bit models | WebGPU kept up to 85% of native decoding speed [17] |
| 2024-10-10 | Llama 3 8B fp16, one H100 | About 30 concurrent users at 100 tokens/s each, about 3,000 tokens/s in total [18] |
| 2024-12-20 | WebLLM paper: 4-bit Llama 3.1 8B and Phi-3.5-mini, MacBook Pro M3 Max, WebLLM in the browser versus native MLC LLM | Decoding at 41.1 versus 57.7 tokens/s (71.2%) and 71.1 versus 89.3 tokens/s (79.6%), summarized by the authors as up to 80% of native performance on the same device [41] |
| 2025-01-07 | Llama 3.1 70B FP8, eight H100, two engines | Prefill-decode disaggregation gave up to 16% lower mean and 28% lower 99th-percentile job completion time than data parallelism [20] |
The October 2023 post added that multi-GPU scaling was not linear, and attributed this to the lack of NVLink in its test machines and to GPUs being under-used during token-by-token decoding.[15] The October 2024 study compared MLCEngine with SGLang v0.3.1.post2 and vLLM v0.6.1.post2 and stated that MLCEngine achieved state-of-the-art performance in low-latency settings; the authors wrote that they expected all of the frameworks to improve over time, and published the commands needed to repeat the benchmark.[18] In the XGrammar announcement of November 2024 the project reported that, with Llama 3 8B on an NVIDIA H100 GPU, its end-to-end structured generation was up to 14 times faster than existing engines on JSON-schema tasks and up to 80 times faster on grammar-guided tasks.[19]
Independent evaluations
Several academic studies have measured MLC LLM against other runtimes.
MELT, a benchmarking study by Stefanos Laskaridis and colleagues accepted at MobiCom 2024, ran models on Android phones, iPhones and Jetson boards with MLC LLM and llama.cpp, which it chose as the most popular frameworks with the widest model and platform support. It found that MLC LLM generally gave higher throughput than llama.cpp, with generation throughput 4% higher on average for GPU execution, but "at the cost of model portability (models need to be compiled per platform)". Within MLC LLM, the Metal-accelerated iPhones averaged 78.93% higher generation throughput than the OpenCL-accelerated Android phones.[47]
A 2025 preprint by Varun Rajesh and colleagues compared MLX, MLC LLM, llama.cpp, Ollama and PyTorch MPS on a Mac Studio with an M2 Ultra, using the Qwen 2.5 family. Under its settings MLX had the highest sustained generation throughput, while MLC LLM had consistently lower time to first token for moderate prompt sizes and, in the authors' words, "stronger out-of-the-box inference features".[48]
Authors of competing mobile engines have used MLC LLM as a baseline. The Transformer-Lite paper (2024) reported more than 10 times the prefill speed and 2 to 3 times the decoding speed of the CPU-based FastLLM and the GPU-based MLC LLM on phone GPUs.[49] The BitLLM paper (October 2026) reported average speedups of 1.78 times over llama.cpp, 1.73 times over MNN-LLM, 1.58 times over LiteRT-LM and 1.47 times over MLC LLM on mobile GPUs.[50]
Limitations
The documentation and the sources above record several constraints.
- Per-platform compilation. A model needs converted weights and a model library compiled for the specific backend. The MELT authors named this loss of portability as the cost of MLC LLM's higher throughput.[31][47]
- Web builds. Compiling for WebGPU is not possible with the prebuilt Python package; it needs a source build and a WebAssembly toolchain.[31] The project's own measurements put WebGPU at up to 85% of native decoding speed in a June 2024 post and up to 80% of native performance in the WebLLM paper.[17][41]
- Backend coverage. The GPU setup page states that only ROCm 6.1 and 6.2 are supported, FP8 quantization is limited to CUDA, and the AWQ mode is marked unstable.[32][34]
- Android. The demo APK is built for the Samsung S23 with a Snapdragon 8 Gen 2 chip, and the quick-start page lists a Redmi Note 12 Pro and Google Pixel phones as the other tested devices. Building for Android under Windows is described as experimental. The documentation also records a known issue on Adreno GPUs in which models in one weight layout freeze the system interface for about 20 to 50 seconds before the first inference.[38][43]
- Memory on phones. The project wrote in May 2023 that iOS enforced a 4 GB RAM limit per app that Android did not have, so its Android build could use more memory than its iOS build.[12]
- Concurrency. The synchronous Python engine does not batch concurrent requests; batching requires the asynchronous class or the server.[35]
- Releases. The package metadata classifies the project's development status as beta, the installation instructions point to nightly wheels, and the GitHub Releases page has no entry later than the 2023 pre-release.[33][55][58]
People and organizations
The 2023 posts state that MLC LLM was initiated by members of CMU Catalyst, UW SAMPL (a University of Washington lab), SJTU (Shanghai Jiao Tong University), OctoML and the MLC community.[5][12] Catalyst is a machine learning and systems research group at Carnegie Mellon University, drawing on its Machine Learning, Computer Science, and Electrical and Computer Engineering departments; it lists MLC LLM, Apache TVM and XGrammar among its research projects.[8][52] The MLC community's site, mlc.ai, describes itself as "An Open Community of ML Compilations" and features MLC LLM, WebLLM and XGrammar among its projects.[54]
Tianqi Chen, the first author of the TVM paper, lists MLC LLM first among the machine learning systems he started, ahead of Apache TVM and XGBoost. His homepage gives his positions as associate professor in Carnegie Mellon's Machine Learning and Computer Science departments and distinguished engineer at NVIDIA, and records that he was chief technologist of OctoAI from 2019 to 2024.[24][51] Carnegie Mellon's Computer Science Department directory also lists him as an associate professor, in the Machine Learning and Computer Science departments.[63]
The 2023 posts credit OctoML and the 2024 posts credit OctoAI; a company press release of April 2024 gives the name as "OctoAI (formerly OctoML)".[5][6][59] GeekWire reported on September 25, 2024 that OctoAI had been acquired by NVIDIA, adding that an NVIDIA spokesperson declined to comment and that OctoAI's homepage carried the line "OctoAI is now NVIDIA".[61] An archived copy of that homepage from 1 October 2024 shows the same heading, and Chen's homepage describes the company as acquired by NVIDIA.[51][53] The repository remains under the mlc-ai organization on GitHub, and its notice file assigns copyright to "MLC LLM Contributors".[4][55]
Status as of November 2026
On 19 November 2026 the repository was not archived. GitHub's API reported 1,828 commits on the main branch, the most recent dated 2026-11-16, and 376 open issues and pull requests.[4][23] Version tags run from v0.1.dev0 (April 2023) through v0.17 and v0.18 (2024) and v0.19.0 (February 2025) to v0.20.0 (July 2026); a further tag, v0.26.dev0, points to a commit dated 2026-06-28. The annotation on the v0.20.0 tag reads "mlc-llm v0.20.0 stable (last monolithic-era release; aligns with mlc-ai 0.20.0)".[22] The package metadata on the main branch gives the development version as 0.20.0.dev0 and lists FlashInfer as a Linux dependency.[55]
Relation to other inference software
Published comparisons place MLC LLM beside two groups of software. On phones and personal computers it is measured against llama.cpp, MLX and Ollama.[47][48] On data-center GPUs the project benchmarks MLCEngine against vLLM and SGLang.[18] The project gives this range as the reason for building one engine: its 2024 announcement argued that techniques from server deployments, such as continuous batching, would become useful on local devices as well.[6]
It differs from llama.cpp in method. MLC LLM generates device-specific code with a compiler and needs a compiled library per platform, whereas llama.cpp reads models in GGUF format.[5][47]
Continue reading
- llama.cppllama.cpp is an open-source large language model inference engine written in C and C++ by Bulgarian software engineer Georgi Gerganov that runs large language models on consumer-grade hardware without…Mentioned 8 times above
- OctoAIOctoAI (originally OctoML) was an American artificial intelligence infrastructure company that operated a generative-AI inference platform and, before its pivotMentioned 7 times above
- MLXMLX is an open-source array and machine-learning framework built by Apple Inc.'s machine-learning research team specifically for Apple silicon, the M1, M2, M3, M4, and M5 series of system-on-chip processors.Mentioned 3 times above
- OpenAI-compatible APIAn OpenAI-compatible API is an HTTP interface, offered by an inference server or a model provider other than OpenAI, that accepts requests and returns responses in the shape of the OpenAI APIListed in See also
- Structured outputStructured output is a set of techniques and API features that constrain a large language model (LLM) to emit responses that exactly conform to a predefined format or schema, such as JSON, XML, or a custom…Listed in See also
- Local AILocal AI is the practice of running artificial intelligence models on hardware that the user owns or controls, such as a laptop, a desktop PC, a workstation or a home serverListed in See also
More from See also
More to read
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11MLC LLM. "Introduction to MLC LLM". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...introduction
- ^1 ^2 ^3mlc-ai. "MLC LLM: Universal LLM Deployment Engine with ML Compilation" (README). GitHub, retrieved November 19, 2026. github.com/...mlc-llm
- ^1 ^2mlc-ai/mlc-llm. "LICENSE" (Apache License, Version 2.0). GitHub, retrieved November 19, 2026. github.com/...LICENSE
- ^1 ^2 ^3 ^4 ^5 ^6 ^7GitHub. "mlc-ai/mlc-llm" (repository metadata). GitHub REST API, retrieved November 19, 2026. api.github.com/...mlc-llm
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8MLC Community. "Bringing Hardware Accelerated Language Models to Consumer Devices". MLC Blog, May 1, 2023. blog.mlc.ai/...ccelerated-llm-to-consumer-hardware
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12MLC Community. "MLC-LLM: Universal LLM Deployment Engine with ML Compilation". MLC Blog, June 7, 2024. blog.mlc.ai/...ployment-engine-with-ML-compilation
- ^1 ^2 ^3 ^4 ^5 ^6MLC LLM. "REST API". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...rest
- ^1 ^2Carnegie Mellon Catalyst Group. "Machine Learning Compilation for Large Language Models". catalyst.cs.cmu.edu, retrieved November 19, 2026. catalyst.cs.cmu.edu/...mlc-llm
- ^GitHub. "mlc-ai/web-stable-diffusion" (repository metadata). GitHub REST API, retrieved November 19, 2026. api.github.com/...web-stable-diffusion
- ^1 ^2 ^3 ^4GitHub. "mlc-ai/web-llm" (repository metadata and latest release). GitHub REST API, retrieved November 19, 2026. api.github.com/...web-llm
- ^mlc-ai/mlc-llm. "Initial commit" (commit cea09f2). GitHub, April 29, 2023. github.com/...9f2f5651fab837109505cb9869ecf6a92b2a
- ^1 ^2 ^3 ^4MLC Community. "Bringing Hardware Accelerated Language Models to Android Devices". MLC Blog, May 8, 2023. blog.mlc.ai/...-language-models-to-android-devices
- ^1 ^2 ^3MLC Community. "Bringing Open Large Language Models to Consumer Devices". MLC Blog, May 22, 2023. blog.mlc.ai/...language-models-to-consumer-devices
- ^1 ^2 ^3 ^4 ^5 ^6MLC Community. "Making AMD GPUs competitive for LLM inference". MLC Blog, August 9, 2023. blog.mlc.ai/...-GPUs-competitive-for-LLM-inference
- ^1 ^2 ^3 ^4 ^5 ^6MLC Community. "Scalable Language Model Inference on Multiple NVIDIA and AMD GPUs". MLC Blog, October 19, 2023. blog.mlc.ai/...nference-on-Multiple-NVDIA-AMD-GPUs
- ^1 ^2 ^3 ^4MLC Community. "GPU-Accelerated LLM on a $100 Orange Pi". MLC Blog, April 20, 2024. blog.mlc.ai/...GPU-Accelerated-LLM-on-Orange-Pi
- ^1 ^2 ^3 ^4 ^5 ^6 ^7MLC Community. "WebLLM: A High-Performance In-Browser LLM Inference Engine". MLC Blog, June 13, 2024. blog.mlc.ai/...nce-in-browser-llm-inference-engine
- ^1 ^2 ^3 ^4 ^5 ^6MLC Community. "Optimizing and Characterizing High-Throughput Low-Latency LLM Inference in MLCEngine". MLC Blog, October 10, 2024. blog.mlc.ai/...hroughput-low-latency-llm-inference
- ^1 ^2 ^3 ^4MLC Community. "Achieving Efficient, Flexible, and Portable Structured Generation with XGrammar". MLC Blog, November 22, 2024. blog.mlc.ai/...structured-generation-with-xgrammar
- ^1 ^2 ^3MLC Community. "Microserving LLM engines". MLC Blog, January 7, 2025. blog.mlc.ai/...microserving-llm-engines
- ^MLC Community. "XGrammar-2: Fast and Customizable Structured Generation for Tool Calling and Agents". MLC Blog, May 4, 2026. blog.mlc.ai/...-customizable-structured-generation
- ^1 ^2 ^3mlc-ai/mlc-llm. "Tags". GitHub, retrieved November 19, 2026. github.com/...tags
- ^1 ^2 ^3 ^4mlc-ai/mlc-llm. "Commits" (main branch). GitHub, retrieved November 19, 2026. github.com/...main
- ^1 ^2 ^3Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, Arvind Krishnamurthy. "TVM: An Automated End-to-End Optimizing Compiler for Deep Learning". 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 578-594, October 2018. usenix.org/...chen
- ^Ruihang Lai, Junru Shao, Siyuan Feng, Steven S. Lyubomirsky, Bohan Hou, Wuwei Lin, Zihao Ye, Hongyi Jin, Yuchen Jin, Jiawei Liu, Lesheng Jin, Yaxing Cai, Ziheng Jiang, Yong Wu, Sunghyun Park, Prakalp Srivastava, Jared G. Roesch, Todd C. Mowry, Tianqi Chen. "Relax: Composable Abstractions for End-to-End Dynamic Machine Learning". Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '25), Volume 2, pp. 998-1013, 2025. arXiv:2311.02103. arxiv.org/...2311.02103
- ^Siyuan Feng, Bohan Hou, Hongyi Jin, Wuwei Lin, Junru Shao, Ruihang Lai, Zihao Ye, Lianmin Zheng, Cody Hao Yu, Yong Yu, Tianqi Chen. "TensorIR: An Abstraction for Automatic Tensorized Program Optimization". Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2023), Volume 2, pp. 804-817, 2023. arXiv:2207.04296. arxiv.org/...2207.04296
- ^MLC LLM. "Install TVM Compiler". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...tvm
- ^1 ^2mlc-ai/mlc-llm. ".gitmodules". GitHub, retrieved November 19, 2026. github.com/....gitmodules
- ^1 ^2MLC LLM. "Define New Model Architectures". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...define_new_models
- ^Siyuan Feng, Jiawei Liu, Ruihang Lai, Charlie F. Ruan, Yong Yu, Lingming Zhang, Tianqi Chen. "Productively Deploying Emerging Models on Emerging Platforms: A Top-Down Approach for Testing and Debugging". arXiv:2404.09151, April 14, 2024. arxiv.org/...2404.09151
- ^1 ^2 ^3 ^4 ^5 ^6MLC LLM. "Compile Model Libraries". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...compile_models
- ^1 ^2MLC LLM. "Configure Quantization". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...configure_quantization
- ^1 ^2MLC LLM. "Install MLC LLM Python Package". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...mlc_llm
- ^MLC LLM. "GPU Drivers and SDKs". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...gpu
- ^1 ^2MLC LLM. "Python API". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...python_engine
- ^MLC LLM. "CLI". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...cli
- ^1 ^2 ^3MLC LLM. "iOS Swift SDK". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...ios
- ^1 ^2 ^3 ^4 ^5MLC LLM. "Android SDK". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...android
- ^1 ^2MLC LLM. "WebLLM Javascript SDK". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...webllm
- ^1 ^2mlc-ai. "WebLLM: High-Performance In-Browser LLM Inference Engine" (README). GitHub, retrieved November 19, 2026. github.com/...web-llm
- ^1 ^2 ^3Charlie F. Ruan, Yucheng Qin, Akaash R. Parthasarathy, Xun Zhou, Ruihang Lai, Hongyi Jin, Yixin Dong, Bohan Hou, Meng-Shiun Yu, Yiyan Zhai, Sudeep Agarwal, Hangrui Cao, Siyuan Feng, Tianqi Chen. "WebLLM: A High-Performance In-Browser LLM Inference Engine". arXiv:2412.15803, December 20, 2024 (revised April 13, 2026). arxiv.org/...2412.15803
- ^Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, Tianqi Chen. "XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models". arXiv:2411.15100 (MLSys '25), November 22, 2024. arxiv.org/...2411.15100
- ^1 ^2 ^3MLC LLM. "Quick Start". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...quick_start
- ^Hugging Face. "mlc-ai" (organization page and model listing). huggingface.co, retrieved November 19, 2026. huggingface.co/mlc-ai
- ^mlc-ai/mlc-llm. "python/mlc_llm/model" (model definitions directory). GitHub, retrieved November 19, 2026. github.com/...model
- ^1 ^2 ^3Apple. "MLC Chat" (App Store listing). apps.apple.com, retrieved November 19, 2026. apps.apple.com/...id6448482937
- ^1 ^2 ^3 ^4 ^5Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, Hamed Haddadi. "MELTing point: Mobile Evaluation of Language Transformers". arXiv:2403.12844 (accepted at MobiCom 2024), March 19, 2024. arxiv.org/...2403.12844
- ^1 ^2Varun Rajesh, Om Jodhpurkar, Pooja Anbuselvan, Mantinder Singh, Ashok Jallepali, Shantanu Godbole, Pradeep Kumar Sharma, Hritvik Shrivastava. "Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS". arXiv:2511.05502, October 9, 2025. arxiv.org/...2511.05502
- ^Luchang Li, Sheng Qian, Jie Lu, Lunxi Yuan, Rui Wang, Qin Xie. "Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs". arXiv:2403.20041, March 29, 2024. arxiv.org/...2403.20041
- ^Hangyu Zheng, Zhihao Shu, Kunxiong Zhu, Minghai Qin, Xiaolong Ma, Gagan Agrawal, Wei Niu. "BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs". arXiv:2611.00547, October 31, 2026. arxiv.org/...2611.00547
- ^1 ^2Tianqi Chen. "Tianqi Chen" (personal homepage). tqchen.com, retrieved November 19, 2026. tqchen.com
- ^Carnegie Mellon Catalyst Group. "Catalyst: Home". catalyst.cs.cmu.edu, retrieved November 19, 2026. catalyst.cs.cmu.edu
- ^OctoAI. "Efficient, Customizable, and Reliable GenAI Inferences" (homepage, archived by the Internet Archive on October 1, 2024). web.archive.org/...octo.ai
- ^1 ^2MLC. "MLC - A Community of Machine Learning Compilers" (home page, headed "An Open Community of ML Compilations"). mlc.ai, retrieved November 19, 2026. mlc.ai
- ^1 ^2 ^3mlc-ai/mlc-llm. "pyproject.toml" and "NOTICE". GitHub, retrieved November 19, 2026. github.com/...pyproject.toml
- ^MLC LLM. "Implement LLM Cross-engine Orchestration Patterns". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...tutorial
- ^1 ^2 ^3MLC LLM. "IDE Integration". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...ide_integration
- ^1 ^2 ^3GitHub. "mlc-ai/mlc-llm releases". GitHub REST API, retrieved November 19, 2026. api.github.com/...releases
- ^OctoAI. "OctoAI Unveils Industry-First Generative AI Production Stack For The Enterprise". PR Newswire, April 2, 2024. prnewswire.com/...ack-for-the-enterprise-302105751
- ^mlc-ai/mlc-llm. "python/mlc_llm/serve/entrypoints/openai_entrypoints.py" (server route definitions). GitHub, retrieved November 19, 2026. github.com/...openai_entrypoints.py
- ^Taylor Soper. "Chip giant Nvidia acquires OctoAI, a Seattle startup that helps companies run AI models". GeekWire, September 25, 2024 (archived by the Internet Archive on September 25, 2024). web.archive.org/...t-helps-companies-run-ai-models
- ^MLC LLM. "Model Artifact Manifest". MLC LLM documentation, retrieved November 19, 2026. llm.mlc.ai/...model_artifact_manifest
- ^Carnegie Mellon University Computer Science Department. "Tianqi Chen" (faculty directory). csd.cmu.edu, retrieved November 19, 2026. csd.cmu.edu/...tianqi-chen
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
1 revision · v1 · 5,769 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent fact-check before publication on November 19, 2026: 168 claims checked against 78 cited and independent sources; 1 material and 11 minor corrections applied before publication.
Cite this page: AI Wiki. "MLC LLM." aiwiki.ai, updated 19 Nov 2026, fact-checked 19 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/mlc_llm
