GGUF
Last reviewed
Sources
19 citations
Review status
Source-backed
Revision
v8 ยท 5,854 words
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation.
Suggest edit
CtrlK
Last reviewed
Sources
19 citations
Review status
Source-backed
Revision
v8 ยท 5,854 words
Add missing citations, update stale details, or suggest a clearer explanation.
GGUF (GPT-Generated Unified Format) is the standard binary file format for storing large language models for local inference, bundling a model's weights, tokenizer, and metadata into a single self-contained file that loads quickly and runs efficiently on consumer CPUs and GPUs. Created by Georgi Gerganov as part of the llama.cpp and ggml ecosystem, GGUF was introduced on August 21, 2023 as the backward-incompatible successor to the earlier GGML, GGMF, and GGJT formats.[1][2] The official specification states the format "is designed to be unambiguous by containing all the information needed to load a model" and "is also designed to be extensible, so that new information can be added to models without breaking compatibility."[1]
GGUF files are natively supported by llama.cpp and by popular inference tools such as Ollama, LM Studio, GPT4All, Jan, and koboldcpp. The format is the dominant vehicle for distributing quantized models: as of early 2026 the Hugging Face Hub hosted roughly 45,000 public GGUF checkpoints, runnable with a single command, alongside first-class platform integration including a metadata viewer, an inference endpoint service, and a JavaScript parser library.[5][19]
Towards the end of September 2022, Georgi Gerganov began work on GGML (originally "Georgi Gerganov's Machine Learning" library), a lightweight C library for tensor operations.[3] GGML was built with strict memory management and multi-threading in mind, and it provided the computational backbone for inference of transformer-based models without heavy dependencies on frameworks like PyTorch or TensorFlow. Gerganov's first widely used GGML application was whisper.cpp, a C/C++ port of OpenAI's Whisper speech-to-text model, released in October 2022.[3]
The original GGML file format stored model weights alongside a flat list of untyped hyperparameters. While functional, this design had significant limitations: adding new metadata fields required changing the loader code, and there was no versioning to distinguish between format revisions.[7]
In March 2023, shortly after Meta released the first LLaMA models, Gerganov created llama.cpp, a pure C/C++ implementation of the LLaMA inference code with no external dependencies.[3] The project's core goal was to let users run large language models on everyday consumer hardware, including laptops and desktops without dedicated GPUs. Before llama.cpp, Gerganov had built whisper.cpp using GGML as the underlying computation library.
As llama.cpp gained popularity, the shortcomings of the original GGML file format became more pressing. The community needed a format that could carry richer metadata, support new quantization schemes without breaking backward compatibility, and work cleanly with memory-mapped I/O.
The model file formats used by the llama.cpp project evolved in four stages:
| Format | Description | Key Characteristics |
|---|---|---|
| GGML | Original format | Unversioned; flat list of untyped hyperparameters; no alignment requirements; identified by magic number 0x67676d6c |
| GGMF | Versioned GGML | Added a version field to the header; only one version (v1) was ever released; same structure as GGML otherwise |
| GGJT | Aligned format | Aligned tensors for mmap support; three versions (v1, v2, v3) with progressively updated quantization schemes |
| GGUF | Current standard | Key-value metadata store; extensible design; self-contained with tokenizer; introduced August 21, 2023 |
GGUF is based on GGJT but replaces the flat hyperparameter list with a structured key-value metadata system. The specification describes this as the central difference: GGUF uses "a key-value structure for the hyperparameters (now referred to as metadata), rather than a list of untyped values," which "allows for new metadata to be added without breaking compatibility with existing models."[1] This single change made it possible to add new fields (architecture details, tokenizer vocabularies, training parameters) without modifying the loader or breaking older models.[1] The GGUF specification was formalized through Pull Request #302 in the ggerganov/ggml repository, authored by philpax.[2]
The GGUF format itself has gone through three internal versions:[1]
| Version | Changes |
|---|---|
| v1 | Initial release with key-value metadata and tensor alignment |
| v2 | Changed most countable values (array lengths, tensor counts) from uint32 to uint64, enabling support for much larger models |
| v3 | Added big-endian support for architectures such as IBM POWER and s390x; current version |
Versions only increment for structural changes to the binary layout. New metadata fields and quantization types can be added without changing the format version.[1]
A GGUF file is a contiguous binary blob divided into four sequential sections: a header, metadata key-value pairs, tensor information, and tensor data.[1] All multi-byte values are stored in little-endian byte order by default, though version 3 of the format introduced support for big-endian files.[1]
The header is a fixed structure at the start of every GGUF file:[1]
| Field | Type | Size | Description |
|---|---|---|---|
| Magic number | uint32 | 4 bytes | 0x46554747 (the ASCII bytes for "GGUF" stored in little-endian order) |
| Version | uint32 | 4 bytes | Format version (currently 3) |
| Tensor count | uint64 | 8 bytes | Total number of tensors stored in the file |
| Metadata KV count | uint64 | 8 bytes | Number of key-value pairs in the metadata section |
The magic number allows any tool to quickly verify whether a file is a valid GGUF file. The version field lets loaders handle structural differences between format revisions.
The metadata section is a flexible key-value store that carries all the information needed to configure and run the model. Keys are hierarchical ASCII strings written in lower_snake_case with dot separators (for example, general.architecture or llama.context_length). Keys can be up to 65,535 bytes long.[1]
The value types supported in metadata are:[1]
| Type ID | Type Name | Description |
|---|---|---|
| 0 | UINT8 | 8-bit unsigned integer |
| 1 | INT8 | 8-bit signed integer |
| 2 | UINT16 | 16-bit unsigned integer |
| 3 | INT16 | 16-bit signed integer |
| 4 | UINT32 | 32-bit unsigned integer |
| 5 | INT32 | 32-bit signed integer |
| 6 | FLOAT32 | 32-bit IEEE 754 float |
| 7 | BOOL | Boolean (1 byte) |
| 8 | STRING | Length-prefixed UTF-8 string |
| 9 | ARRAY | Typed array of values |
| 10 | UINT64 | 64-bit unsigned integer |
| 11 | INT64 | 64-bit signed integer |
| 12 | FLOAT64 | 64-bit IEEE 754 double |
Common metadata keys include general.architecture (the model architecture, such as llama or mistral), general.name (human-readable model name), [arch].context_length (maximum sequence length), [arch].embedding_length (embedding dimensions), [arch].block_count (number of transformer layers), and tokenizer.ggml.model (the tokenizer type used, such as llama for SentencePiece BPE or gpt2 for byte-level BPE).[1] The specification marks general.architecture as required, and general.quantization_version as required whenever any tensor in the file is quantized.[1] The model's chat formatting can also travel inside the file: many models store a Jinja prompt template under the tokenizer.chat_template key, which llama.cpp and other loaders read to apply the correct conversation format automatically.[1]
This metadata design is a major improvement over earlier formats. Because keys are self-describing strings, new metadata fields can be added without changing the format specification or breaking compatibility with tools that do not recognize the new keys.[1]
Following the metadata, the file contains an array of tensor information entries. Each entry records:[1]
| Field | Type | Description |
|---|---|---|
| Name | string | Identifier for the tensor (up to 64 bytes, e.g. blk.0.attn_q.weight) |
| Number of dimensions | uint32 | Dimensionality of the tensor (up to 4 currently) |
| Dimensions | uint64[] | Size along each axis |
| Type | uint32 | The ggml_type enum value indicating the data type or quantization method |
| Offset | uint64 | Byte offset into the tensor data section |
After the tensor info array, the file is padded to a 32-byte alignment boundary (GGUF_DEFAULT_ALIGNMENT).[1] The tensor data section then follows as a contiguous block of binary data containing the actual model weights. Each tensor's offset must be a multiple of the alignment value, which is essential for memory-mapped access. The alignment can be overridden through the general.alignment metadata key, which must be a multiple of 8; when the key is absent, readers assume the default of 32 bytes.[1]
One of GGUF's most important design features is native support for memory-mapped I/O. When a GGUF file is loaded with mmap (the default in llama.cpp), the operating system maps the file directly into the process's virtual address space. Tensor weights can then be accessed through pointers without copying the data into a separate memory buffer.[9]
This approach has several practical benefits:
read() into memory can begin inference almost instantly with mmap.[9]The 32-byte alignment of tensor data in GGUF files is specifically chosen to support efficient mmap behavior. Aligned data avoids partial-page reads and ensures that individual tensors can be accessed without reading adjacent tensors' data. Justine Tunney's 2023 blog post "Edge AI Just Got Faster" demonstrated that mmap-based loading in llama.cpp delivered significant real-world speedups, especially when combined with operating system optimizations like madvise() hints for sequential access patterns.[9]
Quantization is the process of reducing the numerical precision of model weights, trading a small amount of accuracy for significantly lower memory usage and faster inference. GGUF supports a wide range of quantization types, organized into several families: floating-point formats, legacy quantizations, K-quantizations, and I-quantizations (importance-matrix quantization).
| Type | Bits per Weight | Description |
|---|---|---|
| F32 | 32.0 | Standard IEEE 754 single-precision float |
| BF16 | 16.0 | Bfloat16; shortened single-precision format common in training |
| F16 | 16.0 | IEEE 754 half-precision float; common baseline for quality comparison |
These are unquantized formats that preserve full model precision. F16 is the most commonly used starting point for conversion, as it matches the precision at which most models are distributed.
The original quantization methods from the GGML era use simple round-to-nearest schemes with 32-weight blocks. The "_0" variants store only a scale factor (symmetric quantization), while the "_1" variants store both a scale and a minimum value (asymmetric quantization).
| Type | Bits per Weight | Block Size | Formula | Notes |
|---|---|---|---|---|
| Q4_0 | ~4.5 | 32 | w = q * scale | Symmetric; legacy |
| Q4_1 | ~5.0 | 32 | w = q * scale + min | Asymmetric; legacy |
| Q5_0 | ~5.5 | 32 | w = q * scale | Symmetric; legacy |
| Q5_1 | ~6.0 | 32 | w = q * scale + min | Asymmetric; legacy |
| Q8_0 | ~8.5 | 32 | w = q * scale | Near-lossless; still widely used |
| Q8_1 | ~9.0 | 32 | w = q * scale + min | Asymmetric 8-bit; legacy |
These legacy types have been largely superseded by the K-quantization family, which delivers better quality at comparable file sizes.[10]
K-quantizations were introduced in llama.cpp Pull Request #1684 by Iwan Kawrakow (GitHub user ikawrakow), merged on June 5, 2023.[4] They use a hierarchical "super-block" structure that groups multiple smaller blocks together with shared higher-precision scale factors. This approach produces lower quantization error than legacy methods at the same bit rate. The PR demonstrated that, on the original LLaMA-7B, "6-bit quantized perplexity is within 0.1% or better from the original f16 model."[4] The suffixes _S, _M, and _L stand for Small, Medium, and Large, indicating how aggressively quality-sensitive layers receive higher precision.
| Type | Bits per Weight | Super-Block Structure | Weight Formula | Description |
|---|---|---|---|---|
| Q2_K | 2.625 | 16 blocks of 16 weights | w = q * block_scale(4-bit) + block_min(4-bit) | Extreme compression; noticeable quality loss |
| Q3_K_S | 3.4375 | 16 blocks of 16 weights | w = q * block_scale(6-bit) | Small variant; all layers at Q3_K |
| Q3_K_M | ~3.9 | Mixed precision | Mixed | Medium variant; attention layers at higher precision |
| Q3_K_L | ~4.3 | Mixed precision | Mixed | Large variant; more layers kept at higher precision |
| Q4_K_S | 4.5 | 8 blocks of 32 weights | w = q * block_scale(6-bit) + block_min(6-bit) | Small variant; uniform Q4_K across layers |
| Q4_K_M | ~4.9 | Mixed precision | Mixed | Medium variant; widely recommended as the general-purpose default |
| Q5_K_S | 5.5 | 8 blocks of 32 weights | w = q * block_scale(6-bit) + block_min(6-bit) | Small variant; uniform Q5_K |
| Q5_K_M | ~5.7 | Mixed precision | Mixed | Medium variant; excellent quality-to-size ratio |
| Q6_K | 6.5625 | 16 blocks of 16 weights | w = q * block_scale(8-bit) | High quality; moderate compression |
| Q8_K | ~8.5 | 256 weights per block | w = q * block_scale | Used internally for intermediate quantization results |
The "mixed" variants (_M and _L) do not apply a uniform quantization across all layers. Instead, they assign higher-precision types to the most sensitive layers (such as attention output and feed-forward gate projections) while using the named type for less critical layers. For example, Q4_K_M may use Q6_K for half of the attention and feed-forward weight tensors, with Q4_K for the rest. This targeted approach preserves quality in the most impactful layers while still compressing the bulk of the model.[11]
Q4_K_M is widely regarded as the best general-purpose quantization level, offering a good balance between model size, inference speed, and output quality. Q5_K_M is preferred when memory allows, as it preserves more of the original model's capabilities.[11]
I-quantizations represent a newer family that uses an importance matrix to guide the quantization process. The importance matrix is generated by running a calibration dataset through the model to measure how much each weight contributes to the model's output. Weights that matter more receive higher precision. I-quantizations also use non-linear reconstruction and lookup tables, achieving better quality than K-quantizations at the same bit rate, especially at very low bit widths (2-3 bits). Most I-quant types use 256-weight super-blocks.
| Type | Bits per Weight | Description |
|---|---|---|
| IQ1_S | ~1.56 | Ultra-low 1-bit precision with importance scaling |
| IQ1_M | ~1.75 | Slightly higher precision 1-bit variant |
| IQ2_XXS | ~2.06 | 2-bit; extreme compression using learned lookup tables |
| IQ2_XS | ~2.31 | 2-bit; indices into learned lookup tables |
| IQ2_S | ~2.50 | 2-bit importance-matrix-based quantization |
| IQ2_M | ~2.93 | 2-bit; highest quality variant in the IQ2 family |
| IQ3_XXS | ~3.06 | 3-bit with importance scaling |
| IQ3_XS | ~3.50 | 3-bit extended variant |
| IQ3_S | ~3.44 | 3-bit; better quality than Q3_K at similar size |
| IQ3_M | ~3.76 | 3-bit; highest quality in the IQ3 family |
| IQ4_XS | ~4.25 | 4-bit with importance matrix; competes with Q4_K_M at smaller size |
| IQ4_NL | ~4.50 | 4-bit with non-linear mapping; uses 32-weight blocks |
I-quantizations tend to deliver better quality per byte than K-quantizations at the same bit rate.[11] However, they require an importance matrix file generated by the llama-imatrix tool, which adds an extra step to the quantization workflow. For best results, the calibration dataset should be representative of the model's intended use case.
Recent additions to the GGUF type system include:
| Type | Description |
|---|---|
| TQ1_0 | Ternary quantization; weights restricted to {-1, 0, +1}; for models like BitNet |
| TQ2_0 | Ternary quantization variant |
| MXFP4 | 4-bit Microscaling Block Floating Point; a newer standard for efficient computation |
The two ternary types were merged into llama.cpp in Pull Request #8151 by the contributor compilade to support TriLM and BitNet b1.58 style models, whose weights are restricted to the values -1, 0, and +1. TQ1_0 packs weights at 1.6875 bits each by encoding five ternary digits in a single byte (3 to the power of 5 is 243, which fits in the 256 values of a byte), while TQ2_0 uses 2.0625 bits per weight in exchange for faster decoding; the two are lossless conversions of each other and carry no measured quality difference for genuinely ternary models.[15] MXFP4 (Microscaling FP4) became prominent in August 2025 when it was adopted as the native weight format of OpenAI's gpt-oss models, and it is described in more detail in the developments section below.[14]
Selecting the right quantization type involves balancing model quality, file size, memory usage, and inference speed. As a practical guide:
| Use Case | Recommended Types | Approximate Size (7B model) |
|---|---|---|
| Maximum quality | F16, Q8_0 | 13-14 GB |
| High quality, moderate compression | Q6_K, Q5_K_M | 5.5-7 GB |
| Balanced quality and size | Q4_K_M | ~4.6 GB |
| Aggressive compression, limited RAM | Q3_K_M, IQ3_M | 3.3-3.5 GB |
| Extreme compression | Q2_K, IQ2_M | 2.5-3 GB |
Perplexity measurements on Llama 2 7B illustrate the quality gradient: F16 achieves a perplexity of 7.4924, Q8_0 scores 7.4933 (essentially identical), Q4_K_M scores 7.5692 (minimal degradation), and Q2_K scores 8.6501 (noticeable but functional degradation).[11]
Converting a model from a training framework like PyTorch to GGUF format is typically a two-step process handled by tools included in the llama.cpp repository.[12]
The primary conversion script is convert_hf_to_gguf.py, which reads a model stored in Hugging Face Transformers format (typically SafeTensors files plus a tokenizer and config) and writes a GGUF file at full or reduced precision.[12]
A basic conversion command:
python convert_hf_to_gguf.py ./models/my-model/ --outfile my-model-f16.gguf --outtype f16
The --outtype flag controls the initial precision. Common choices are f16 (16-bit float, lossless relative to typical model distributions), bf16, f32 (32-bit float), and q8_0 (8-bit quantized in one step). The script handles architecture detection, tokenizer extraction, and tensor layout automatically for all supported model families.[12]
Once a model has been converted to a full-precision GGUF file, the llama-quantize tool (compiled as part of llama.cpp) reduces the precision to any of the supported quantization types:[6]
./llama-quantize my-model-f16.gguf my-model-Q4_K_M.gguf Q4_K_M
For improved quality at low bit widths, an importance matrix can be provided:
./llama-imatrix -m my-model-f16.gguf -f calibration-data.txt -o imatrix.dat
./llama-quantize --imatrix imatrix.dat my-model-f16.gguf my-model-IQ3_M.gguf IQ3_M
The importance matrix is generated by running a calibration dataset through the model using the llama-imatrix tool. This extra step is highly recommended for IQ-series quantizations and is also beneficial for K-quantizations at Q3 and below.[11]
For users who prefer not to install llama.cpp locally, Hugging Face provides a web-based tool at ggml-org/gguf-my-repo that can convert and quantize models directly on Hugging Face infrastructure.[5] Users select a source model repository and a target quantization type, and the tool produces a GGUF file hosted on Hugging Face.
Very large models (70B parameters and above) can produce GGUF files that exceed the upload limits of hosting platforms like Hugging Face (50 GB per file). The llama-gguf-split tool addresses this by splitting a GGUF file into multiple shards at the tensor level, never cutting a tensor in half:
llama-gguf-split --split --split-max-size 20G my-model.gguf my-model-split
Each shard retains a complete copy of the header and metadata, so any shard can be inspected independently. The shards can be reassembled with:
llama-gguf-split --merge my-model-split-00001-of-00004.gguf my-model-merged.gguf
llama.cpp's model loader can also read split GGUF files directly without merging them first.
The GGUF specification defines an optional but recommended naming convention so that a file name alone communicates the model, its size, its version, and its encoding. The pattern is <BaseName><SizeLabel><FineTune><Version><Encoding><Type><Shard>.gguf, where the base name identifies the architecture, the size label gives the parameter class (for example 8x7B for a mixture-of-experts model), the version uses a v<Major>.<Minor> form that defaults to v1.0 when omitted, and the shard suffix takes the form <ShardNum>-of-<ShardTotal> with the shard number zero-padded to five digits beginning at 00001. At a minimum a compliant file carries the base name, the size label, and the version so that it can be validated automatically.[1]
GGUF has become the standard file format for local LLM inference. The following tools and libraries support GGUF files natively.
llama.cpp is the reference implementation and the project where GGUF originated.[3] Written in C/C++ with minimal dependencies, it supports inference on x86 (with AVX, AVX2, AVX-512 SIMD optimizations), ARM (with NEON), Apple Metal, CUDA, HIP (AMD), Vulkan, and SYCL backends. llama.cpp reads GGUF files directly and uses memory-mapped I/O for efficient model loading. It also supports hybrid CPU/GPU inference, where some transformer layers run on the GPU and others on the CPU, controlled by the --n-gpu-layers parameter.
Ollama wraps llama.cpp in a user-friendly command-line tool with a built-in model registry. Users can pull models by name (e.g., ollama run llama3), and Ollama handles downloading the appropriate GGUF file, selecting a quantization level, and configuring GPU offloading. It exposes an OpenAI-compatible REST API for integration with other applications. Ollama uses GGUF as its internal model storage format, wrapping GGUF files in its own Modelfile configuration layer, and can run any of the roughly 45,000 GGUF checkpoints on Hugging Face directly with a single ollama run command.[19]
LM Studio is a desktop application with a graphical interface for browsing, downloading, and running GGUF models. It connects directly to the Hugging Face model hub, provides one-click downloads, and includes a chat interface for testing. LM Studio also offers a local server mode that exposes an OpenAI-compatible API, typically on port 1234.
GPT4All, developed by Nomic AI, is a desktop application that prioritizes accessibility and privacy for local model inference. It supports GGUF files and provides a simplified interface for users who want to run language models without technical barriers. GPT4All also includes a local document question-answering feature that can index files on the user's machine.
For programmatic access, two Python libraries provide bindings to GGUF-based inference:
from_pretrained method.Additional tools that support GGUF include Jan (an open-source desktop client), koboldcpp (a llama.cpp fork with a web UI focused on creative writing), text-generation-webui (a popular web interface for LLM inference), and Unsloth (which can export fine-tuned models directly to GGUF). The Wolfram Language added native GGUF import support, and the Hugging Face Transformers library can also load GGUF models via its from_pretrained interface.
Hugging Face has become the primary distribution platform for GGUF models. As of early 2026 the platform hosted roughly 45,000 public GGUF checkpoints, and its web interface includes built-in tools for working with the format.[5][19]
The Hugging Face Hub includes a viewer for GGUF files that displays metadata key-value pairs and tensor information (name, shape, precision) directly on the model page.[5] This viewer allows users to inspect a model's architecture, context length, quantization details, and tokenizer configuration without downloading the file.
Hugging Face maintains the @huggingface/gguf npm package, a JavaScript parser that can read GGUF metadata and tensor information from remotely hosted files:[5]
import { gguf } from "@huggingface/gguf";
const { metadata, tensorInfos } = await gguf(
"https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf"
);
This enables web applications to inspect GGUF models hosted on the Hub without downloading the full file.
Hugging Face Inference Endpoints supports GGUF models out of the box.[5] Users can deploy any GGUF model as a dedicated endpoint with a few clicks, selecting their preferred hardware configuration.
A vibrant community of quantizers publishes pre-quantized GGUF models on Hugging Face. TheBloke (Tom Jobbins) was one of the earliest and most prolific contributors, publishing hundreds of models in multiple quantization variants starting in mid-2023. Other active quantization groups include bartowski (known for rapid turnaround on newly released models), QuantFactory, and lmstudio-community. These repositories typically provide each model in several quantization levels (from Q2_K through Q8_0 and various IQ types), along with benchmarks and recommendations in the model card.
Users can browse all GGUF models on Hugging Face by filtering with the gguf library tag at hf.co/models?library=gguf.[5]
GGUF occupies a specific niche in the ecosystem of model file formats. Understanding how it compares to alternatives helps clarify when to use each format.
| Feature | GGUF | SafeTensors | ONNX | Pickle (.bin) |
|---|---|---|---|---|
| Primary use case | Quantized local inference | Training, fine-tuning, GPU serving | Cross-framework deployment | Legacy PyTorch serialization |
| Stores weights | Yes | Yes | Yes | Yes |
| Stores tokenizer | Yes | No (separate files) | No (separate files) | No (separate files) |
| Stores computation graph | No | No | Yes | Yes (via Python objects) |
| Stores metadata | Yes (rich key-value) | Minimal (tensor names/shapes) | Yes (graph-level) | Python object state |
| Built-in quantization | Yes (1-bit to 8-bit, plus FP16/BF16) | No | Limited (via ONNX Runtime tools) | No |
| Memory mapping | Yes | Yes (zero-copy) | Varies by runtime | No |
| Security | Safe (no code execution) | Safe (no code execution) | Safe (no code execution) | Unsafe (arbitrary code execution via pickle) |
| Self-contained | Yes (single file) | No (multiple files needed) | Yes (graph + weights) | No (needs separate files) |
| Primary ecosystem | llama.cpp, Ollama, LM Studio | Hugging Face Transformers, vLLM | ONNX Runtime, TensorRT, OpenVINO | PyTorch |
SafeTensors is the default format for the Hugging Face Transformers library and is widely used for training, fine-tuning, and GPU-based serving. It stores only tensor data with minimal metadata; the model architecture, tokenizer, and configuration live in separate files. SafeTensors supports zero-copy loading and is safe against arbitrary code execution. However, SafeTensors does not include built-in quantization. Running a 7B parameter model from SafeTensors at FP16 requires roughly 14 GB of VRAM, while a Q4_K_M GGUF of the same model fits in approximately 4.6 GB of RAM.
ONNX (Open Neural Network Exchange) stores the full computation graph of a model alongside its weights, making ONNX files portable across frameworks (PyTorch, TensorFlow, MXNet) and hardware targets (CPUs, GPUs, NPUs, custom accelerators). ONNX is better suited for deployment pipelines that require cross-platform compatibility or hardware-specific optimizations through ONNX Runtime, TensorRT, or OpenVINO. GGUF, by contrast, is optimized for the specific case of running transformer-based LLMs on consumer hardware with aggressive quantization.
Python's pickle format was historically the default serialization method for PyTorch models (stored as .bin files). Pickle files can execute arbitrary Python code during deserialization, which poses a serious security risk when loading models from untrusted sources. Both GGUF and SafeTensors were designed in part to eliminate this vulnerability. Pickle also does not support mmap-based loading and requires the full file to be loaded into memory.
GGUF is not limited to a single model family. The convert_hf_to_gguf.py script in the llama.cpp repository supports conversion for a broad and growing range of transformer-based architectures, including:
New architectures are added regularly by community contributors through pull requests to the llama.cpp repository.
While GGUF is primarily associated with language models, the format can store any collection of tensors with metadata. Projects like stable-diffusion.cpp have used GGUF for diffusion models, though this remains less common than its application in the language model space.
The GGUF format is developed as part of the broader ggml ecosystem. In 2023, Georgi Gerganov founded ggml.ai, a company dedicated to supporting the development of the GGML library and related tools. In February 2026, the ggml.ai team joined Hugging Face, bringing the core llama.cpp team under the Hugging Face umbrella while continuing to develop the project as open-source software.[8]
The llama.cpp GitHub repository (under the ggml-org organization) serves as the central hub for GGUF format development.[3] Format changes are proposed and discussed through GitHub issues and pull requests, and the specification document is maintained alongside the source code at docs/gguf.md.[1]
The community around GGUF and local LLM inference is active across several platforms. The llama.cpp GitHub Discussions board hosts technical conversations about quantization methods, performance optimization, and new model support. Reddit communities such as r/LocalLLaMA serve as gathering places for users sharing benchmark results, hardware recommendations, and quantization comparisons.
GGUF's growth has been driven by the broader trend toward running AI models locally rather than relying on cloud APIs. Privacy concerns, cost reduction, offline availability, and the desire for customization have all contributed to demand for efficient local inference, and GGUF has positioned itself as the format that makes this practical on everyday hardware.
GGUF and its surrounding tooling continued to evolve quickly through 2025 and into 2026.
Native 4-bit models and MXFP4. On August 5, 2025, OpenAI released its first open-weight models since GPT-2, gpt-oss-120b and gpt-oss-20b, under the Apache 2.0 license.[13] The two models carry 116.8 billion and 20.9 billion parameters respectively, and rather than being trained in higher precision and quantized afterward, their mixture-of-experts weights (more than 90 percent of the parameter count) were trained natively in the Microscaling FP4 (MXFP4) format at 4.25 bits per parameter, which lets the 120B model run on a single 80 GB GPU and the 20B model run within 16 GB of memory.[13] On the same day, the llama.cpp team announced native MXFP4 support across all major ggml backends, including CUDA, Vulkan, Metal, and CPU, and described the moment as the beginning of the era of natively trained 4-bit local models.[14] Because GGUF could carry the MXFP4 tensor type and the model's metadata without any structural change to the format, gpt-oss GGUF files appeared on Hugging Face within hours of release.
Ternary quantization in the mainline. Support for ternary models, whose weights take only the values -1, 0, and +1, was merged into llama.cpp through Pull Request #8151, adding the TQ1_0 and TQ2_0 types described above for TriLM and BitNet b1.58 architectures.[15] These types push the lower bound of GGUF quantization below two bits per weight while remaining decodable on commodity CPUs.
Multimodal and platform reach. On April 10, 2025, llama.cpp introduced libmtmd, a multimodal abstraction layer that revived and unified support for vision and audio models; inputs such as images, PDFs, and audio are tokenized and routed through the model's encoder, and many of these projectors ship as companion GGUF files (commonly labeled with an mmproj prefix).[17] On December 17, 2025, the project added full hardware acceleration on Android and ChromeOS devices through a new GUI binding, broadening where GGUF models can run.[17]
Hugging Face stewardship. On February 20, 2026, Hugging Face announced that the ggml.ai team, including Georgi Gerganov and core contributors Xuan-Son Nguyen and Aleksander Grygier, had joined the company.[16] The announcement stated that "the project will continue to be 100% open-source and community driven as it is now," that Gerganov and team would keep dedicating 100 percent of their time to llama.cpp with full autonomy over technical direction, and that a goal of the move was to make it nearly single-click to ship new models from the Transformers library into llama.cpp.[16] The team framed the shared objective as providing "the community with the building blocks to make open-source superintelligence accessible to the world over the coming years."[16]
Dynamic and per-layer quantization. As model releases accelerated, the most prolific early quantizer, TheBloke, stopped publishing new files around 2024, and contributors such as bartowski and the Unsloth team became the leading sources of community GGUF builds.[18] Unsloth introduced a method it calls Dynamic 2.0 GGUFs, which analyzes each layer of a model and selects a per-layer quantization type rather than applying one uniform bit width, keeping sensitive embedding and early or late attention layers at higher precision while compressing the more redundant middle feed-forward layers more aggressively; the company reports lower perplexity and KL divergence against the full-precision model than standard importance-matrix quantization.[18]