Citation and evidence

FP8

28 min full readUpdated 45 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

AI ComputeAI HardwareAI InferenceTraining & Optimization

Cite this article

FP8 is a family of 8-bit floating-point number formats used to accelerate deep learning training and inference, built around two encodings, E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2 mantissa bits), that NVIDIA, Arm, and Intel proposed jointly in September 2022 [1][2]. An FP8 value occupies a single byte, half the size of FP16 or bfloat16, which halves memory and bandwidth requirements and, on most accelerators, lets matrix units deliver roughly twice the arithmetic throughput of 16-bit math. FP8 became practical at scale with NVIDIA's Hopper generation and its Transformer Engine, and is now supported across accelerators from AMD, Intel, Google, and others. It is used both for training, most prominently in DeepSeek-V3's FP8 mixed-precision framework, and as a mainstream quantization format for serving large language models. Because the format's range is narrow, every FP8 tensor travels with higher-precision scale factors, and the choice of scaling granularity (per tensor, per channel, per token, or per block) is the main design decision in any FP8 recipe. The Open Compute Project's Microscaling (MX) formats extend FP8 with block-level scaling and connect it to its 4-bit successor, FP4. The 2022 specification paper reported that FP8 training was "effectively matching the result quality achieved by 16-bit training sessions" across CNNs, RNNs, and Transformer models, including language models up to 175 billion parameters [1].

What problem does FP8 solve?

Eight-bit floating-point arithmetic predates the standardization push. IBM researchers demonstrated FP8 training in 2018 using a 1-5-2 (sign-exponent-mantissa) format with chunk-based higher-precision accumulation [3], and in 2019 proposed a hybrid scheme (HFP8) that used a 1-4-3 format with an adjusted exponent bias for the forward pass and a 1-5-2 format for the backward pass, prefiguring the later E4M3/E5M2 split [26]. Tesla's Dojo whitepaper described two configurable 8-bit formats, CFloat8_1_4_3 and CFloat8_1_5_2, whose exponent bias is set in software rather than fixed [27], and Graphcore researchers published a study of 8-bit numerical formats for deep neural networks in June 2022 [28].

In September 2022, NVIDIA, Arm, and Intel published "FP8 Formats for Deep Learning", an interchange specification built around the E4M3 and E5M2 encodings. The paper reported that FP8 training effectively matched 16-bit result quality across CNNs, RNNs, and Transformer models, including GPT-3-style language models up to 175 billion parameters [1][2]. Those training experiments used simulated FP8: tensors were clipped to FP8-representable values, while the arithmetic itself ran in FP16 or bfloat16 [1]. Two months earlier, in July 2022, Graphcore had proposed its own 8-bit floating-point format to the IEEE working group defining machine-learning arithmetic, with AMD advocating its adoption and Qualcomm expressing support [29]. The Graphcore encodings keep a single zero and a single NaN (the "FNUZ" variants), and they later appeared in AMD hardware [4][31]. The Open Compute Project codified the NVIDIA/Arm/Intel encodings as the OCP 8-bit Floating Point Specification (OFP8), revision 1.0, approved on June 20, 2023 with contributions from AMD, Arm, Google, Intel, Meta, and NVIDIA; a December 2023 revision corrected the exponent biases listed in one section [5]. The IEEE P3109 working group is developing a draft standard that defines a parameterized family of narrow binary floating-point formats for machine learning, varying width, precision, signedness, and the presence of infinities [30].

Original eight-bit field diagrams for E4M3 and E5M2, distinguishing sign, exponent and fraction bits by label and color.

The E4M3 and E5M2 encodings both occupy eight bits: one sign bit, plus four or five exponent bits and three or two trailing significand (fraction) bits. The diagram shows the field allocation, not every special-value rule or a tensor scaling scheme.[25] Illustration: AI Wiki original diagram. CC0 1.0.

What are the E4M3 and E5M2 formats?

Both encodings use one sign bit and split the remaining seven bits between exponent and mantissa:

  • E4M3 (4 exponent bits, 3 mantissa bits, bias 7) favors precision. To stretch its range it departs from IEEE 754 conventions: it has no infinities and reserves only two bit patterns for NaN, which frees the top exponent values and extends the largest finite magnitude to 448 [1].
  • E5M2 (5 exponent bits, 2 mantissa bits, bias 15) favors range and follows IEEE 754 conventions for infinities and NaNs. It is effectively IEEE FP16 with the mantissa truncated to two bits, making conversion between the two trivial [1].
PropertyE4M3E5M2
Bit layout (sign/exponent/mantissa)1/4/31/5/2
Exponent bias715
Largest finite value44857,344
Smallest normal value2^-6 (approx. 0.016)2^-14 (approx. 6.1e-5)
Largest subnormal value0.875 x 2^-60.75 x 2^-14
Smallest subnormal value2^-9 (approx. 0.002)2^-16 (approx. 1.5e-5)
Dynamic range18 binades32 binades
InfinitiesNonePositive and negative
NaN bit patterns2 (S.1111.111)6 (S.11111.01, .10, .11)
Typical roleWeights, activationsGradients

Expanded article table

The values above follow the 2022 paper and the OFP8 specification [1][5]. The tradeoff is range versus precision: E4M3 spans about 18 binades (powers of two) with a 3-bit mantissa, while E5M2 spans about 32 binades with only a 2-bit mantissa [1]. The 2022 paper states that "the recommended use of FP8 encodings is E4M3 for weight and activation tensors, and E5M2 for gradient tensors" [1]: weight and activation distributions are comparatively narrow, while gradients vary over a much wider range. Because even E5M2's range is tiny next to FP32, FP8 tensors are almost always paired with higher-precision scaling factors chosen so that a tensor's values fill the representable range; how those scales are computed, and at what granularity, is the central design question in both FP8 training and inference.

The paper explains the E4M3 compromise in detail. Dropping infinities and all but one mantissa pattern for NaN adds seven magnitudes (256, 288, 320, 352, 384, 416, and 448) and one extra binade; without that change the maximum would have been 240. The authors declined to go further: a single zero and a single NaN would have added one more magnitude, 480, but they kept both signs because positive and negative encodings preserve IEEE properties such as comparing and sorting values with integer operations [1]. They also kept IEEE-style exponent biases and left range placement to per-tensor scale factors in software, noting that a programmable exponent bias is equivalent to allowing only power-of-two scale factors [1].

Conversion, rounding, and overflow

The OFP8 specification defines conversion from FP32, FP16, and bfloat16. Implementations must support round-to-nearest-even and both a saturating and a non-saturating mode, and are not required to raise exception flags [5]:

Source value (after rounding)E4M3, saturatingE4M3, non-saturatingE5M2, saturatingE5M2, non-saturating
NaNNaNNaNNaNNaN
Plus or minus infinityPlus or minus 448NaNPlus or minus 57,344Plus or minus infinity
Magnitude above the format maximumPlus or minus 448NaNPlus or minus 57,344Plus or minus infinity
Magnitude below the smallest subnormalSigned zeroSigned zeroSigned zeroSigned zero

Expanded article table

The 2022 paper recommends saturating overflowing values to the largest representable magnitude. It argues that the FP16 automatic-mixed-precision practice of skipping a weight update on overflow is a poor fit for FP8, because overflows are much more likely with the narrower range and too many updates would be skipped [1].

Vendor variants (FNUZ)

AMD's CDNA 3 accelerators (the MI300 series) implement variant "FNUZ" encodings (finite and NaN only, unsigned zero): there are no infinities, a single NaN occupies the negative-zero bit pattern, and the exponent biases are 8 and 16 rather than 7 and 15 [4][31]. As a result E4M3FNUZ tops out at 240 instead of 448, and an OCP E4M3 value (called E4M3FN in ONNX) cannot be turned into E4M3FNUZ by reusing its bits [31]. ONNX added all four types, E4M3FN, E4M3FNUZ, E5M2, and E5M2FNUZ, in version 1.15 [31]. The mismatch complicates moving FP8 checkpoints between vendors. AMD's HIP documentation lists FNUZ as the hardware FP8 type on CDNA 3 and the OCP encodings on CDNA 4 (the MI350 series, 2025) and on RDNA 4 [32].

Which hardware supports FP8?

NVIDIA's Hopper architecture, announced in March 2022, introduced FP8 tensor cores: the H100 delivers close to 4 petaFLOPS of FP8 compute (3,958 TFLOPS with structured sparsity, 1,979 TFLOPS dense), twice its FP16 tensor rate [45]. Hopper paired the format with the Transformer Engine, a combination of tensor core hardware and library software that chooses between FP8 and 16-bit precision per layer and manages scaling and recasting automatically [6][7]. In the open-source Transformer Engine library, per-tensor scales are derived either from a history of recent absolute-maximum (amax) values ("delayed scaling") or computed just in time ("current scaling"); delayed scaling is faster but can hurt convergence, and NVIDIA's later guidance favors current scaling and finer-grained per-block recipes [7][8]. The Ada Lovelace generation (L4, L40S) also includes FP8 tensor cores, and vLLM supports FP8 weight-and-activation computation on GPUs of compute capability 8.9 and above (Ada Lovelace, Hopper, Blackwell) [15]. Blackwell (2024) adds a second-generation Transformer Engine with native block-scaled microscaling support, including MXFP8 [7][9].

AMD's Instinct MI300X (launched December 2023) reaches a peak theoretical 2,614.9 teraFLOPS of dense FP8 [10], using the FNUZ encodings described above [4]. The CDNA 4-based MI355X switched to the OCP encodings and lists 5 petaFLOPS of dense OCP FP8 (E5M2 and E4M3) and the same 5 petaFLOPS for MXFP8 [32][40]. Intel's Gaudi 2 supports FP8 in both E5M2 and E4M3, with delayed scaling as an option; enabling FP8 roughly doubled Gaudi 2's speed on the MLPerf GPT-3 training benchmark in November 2023, cutting time to train to 153.58 minutes on 384 accelerators [11]. Gaudi 3 is rated at 1,835 teraFLOPS of dense FP8, the same figure Intel gives for its FP16 and BF16 throughput [39]. Google compares its TPU generations by peak FP8 FLOPS and says FP8 is natively supported on its seventh-generation TPU, Ironwood (4,614 teraFLOPS peak per chip), whereas the FP8 figures for TPU v4 and v5p are emulated [42]. AWS quotes its Trainium3 accelerator in MXFP8 terms, up to 362 MXFP8 petaFLOPS for a 144-chip UltraServer [43].

AcceleratorArchitectureFP8 encodingsVendor peak FP8 matrix throughput
NVIDIA H100 SXMHopperE4M3, E5M21,979 TFLOPS dense; 3,958 with sparsity [45]
AMD Instinct MI300XCDNA 3E4M3FNUZ, E5M2FNUZ2,614.9 TFLOPS dense; 5,229.8 with sparsity [10][32]
Intel Gaudi 3Gaudi 3FP81,835 TFLOPS dense [39]
NVIDIA HGX B200 (8 GPUs)BlackwellE4M3, E5M2, MXFP872 PFLOPS FP8/FP6 with sparsity; dense is half [41]
NVIDIA HGX B300 (8 GPUs)Blackwell UltraE4M3, E5M2, MXFP872 PFLOPS FP8/FP6 with sparsity; dense is half [41]
AMD Instinct MI355XCDNA 4OCP E4M3, E5M2; MXFP85 PFLOPS dense; 10.1 with sparsity [40]
NVIDIA Rubin GPURubinFP8/FP617.5 PFLOPS dense (NVIDIA listing, training) [41]
Google Ironwood TPUTPU, seventh generationFP8 (native)4,614 TFLOPS peak per chip [42]

Expanded article table

How are FP8 scale factors chosen?

Every FP8 tensor is stored with at least one higher-precision scale factor. Before casting, values are multiplied by a scale chosen so that the tensor's largest magnitude (its amax) lands near the format maximum; after a matrix multiply, the output is unscaled once per dot product, which the 2022 paper notes costs little because it is amortized over many multiply-accumulates [1]. NVIDIA's Transformer Engine documentation explains why the single loss-scaling factor used in FP16 training is not enough: FP8's range can hold any one activation or gradient tensor, but not all of them at once, so each FP8 tensor needs its own scale [7]. Two choices define a recipe: how many values share a scale (granularity), and when the amax is measured.

GranularityValues sharing one scaleScale typeTypical use
Per tensorWhole tensorFP32Original Hopper FP8 training recipes [7]
Per channel (row-wise)One output channel or rowFP32Weight quantization for inference [1][15]; Meta's Llama 3 405B FP8 inference [33]
Per tokenOne token's activation vectorFP32Dynamic activation quantization in vLLM [15]
1x128 tileOne token across 128 channelsFP32 in Transformer EngineDeepSeek-V3 activations [14][8]
128x128 blockA 128 by 128 weight blockFP32 in Transformer EngineDeepSeek-V3 weights and released checkpoint [14][38]
MXFP8 block32 consecutive valuesE8M0 (power of two)Hardware-native on Blackwell and MI355X [7][17][40]

Expanded article table

Finer granularity reduces the damage a single outlier can do. NVIDIA notes that different attention heads in one weight matrix can have different magnitudes and that activations from SwiGLU layers or mixture-of-experts routers mix dense clusters of small values with rare extreme outliers, so a single per-tensor scale either sacrifices precision for the small values or clips the outliers [8]. DeepSeek found that its tile-wise and block-wise grouping let it use E4M3 for every tensor, because the small groups effectively share exponent bits [14].

Timing matters as much as granularity. Delayed scaling sets the scale from a history of amax values recorded in previous iterations, which allows full-speed FP8 kernels but requires storing that history; scaling from the amax of the tensor being produced requires writing the output in higher precision, scanning it, and then casting it [7]. NVIDIA's 2025 guidance notes that outliers can dominate an amax history and destabilize training, and presents per-tensor current scaling, which measures amax in the current pass, as simpler and more robust to outliers [8]. DeepSeek-V3 likewise rejected delayed quantization and computed the amax online for each tile and block [14].

Transformer Engine recipeNeMo fp8_recipe valueScaling
DelayedScalingdelayedPer tensor, from an amax history; "HYBRID" format uses E4M3 forward and E5M2 backward [7][8]
Float8CurrentScalingtensorwisePer tensor, from the current amax [7][8]
Float8BlockScalingblockwiseConfigurable blocks such as 1x128 or 128x128, as in DeepSeek-V3 [7][8]
MXFP8BlockScalingmxfp832-element blocks with E8M0 scales, E4M3 for all tensors [7][8]
NVFP4BlockScalingNot listed16-element blocks with E4M3 scales plus a per-tensor FP32 scale [7]

Expanded article table

Block scaling has a cost that per-tensor scaling does not: an MXFP8 tensor and its transpose are not numerically identical, because the scale runs along one direction. Blackwell tensor cores need MXFP8 data to be contiguous along the reduction dimension, so Transformer Engine quantizes both a regular and a transposed copy directly from the high-precision input rather than transposing already-quantized data [7].

How does FP8 training work?

FP8 training is a form of mixed-precision training: matrix multiplications take FP8 inputs while master weights, optimizer states, and numerically sensitive operations (normalization, attention softmax, embeddings, output projections) remain in BF16 or FP32, and accumulation inside the matrix units happens at higher precision. The standard recipe stores weights and activations as E4M3 and gradients as E5M2, with one scale factor per tensor [1][7].

Early large-scale adopters included Inflection AI, which trained Inflection-2 (announced November 2023) on 5,000 H100 GPUs in FP8 mixed precision for roughly 10^25 floating-point operations [12]. Microsoft's FP8-LM framework (October 2023) pushed FP8 beyond the matrix multiplies into gradient communication and optimizer state, reporting 39 percent lower real memory usage and 75 percent faster training than BF16 Megatron-LM on a GPT-175B configuration [13].

The most influential demonstration came with DeepSeek-V3 (December 2024), a 671-billion-parameter mixture-of-experts model trained on 14.8 trillion tokens in 2.788 million H800 GPU-hours. DeepSeek's framework runs all three linear-layer matrix multiplications (forward, activation gradient, and weight gradient) in FP8 and, unusually, uses E4M3 everywhere rather than the E4M3/E5M2 hybrid [14]. It compensates for the narrow range with fine-grained scaling: activations are scaled per 1x128 tile (one token by 128 channels) and weights per 128x128 block. Because the team found that FP8 matrix-multiply accumulation on H800 tensor cores retains only around 14 bits of precision, partial sums are promoted to FP32 registers on the CUDA cores at intervals of 128 elements [14]. The report's FP8-versus-BF16 comparison, run on two smaller MoE baselines (about 16 billion parameters trained on 1.33 trillion tokens, and about 230 billion parameters trained on about 0.9 trillion tokens), kept the relative loss error below 0.25 percent, and DeepSeek describes its framework as validating FP8 training "for the first time" on an extremely large-scale model [14]. To save memory, DeepSeek also caches activations in FP8 and tracks the AdamW moments in BF16, while master weights and the gradients used for batch accumulation stay in FP32 [14]. A few activations get special treatment: inputs to the linear layer after attention use a custom E5M6 format, and some activations use scale factors restricted to integral powers of two [14]. The released checkpoint is itself stored natively in FP8: its configuration specifies E4M3 weights with 128x128 block scales and dynamic activation quantization [38].

On Blackwell, block-scaled MXFP8 removes the need for the E4M3/E5M2 split. NVIDIA reports MXFP8 validation-loss curves that track BF16 for 2-billion- and 8-billion-parameter Nemotron models [8].

Which layers are usually kept in higher precision?

FP8 recipes rarely quantize every layer. Operations that are numerically sensitive, or so cheap that higher precision costs almost nothing, are commonly left in BF16 or FP32:

RecipeKept in higher precisionSource
DeepSeek-V3 FP8 trainingEmbedding module, output head, MoE gating modules, normalization operators, attention operators; master weights, weight gradients, and optimizer states[14]
Meta Llama 3 405B FP8 inferenceSelf-attention parameters, plus the first and last Transformer layers; FP8 was applied to most feed-forward parameters and activations[33]
vLLM / LLM Compressor FP8_DYNAMIC exampleThe lm_head (ignore=["lm_head"]); vLLM's online fp8_per_tensor mode also skips the final lm_head[15]
vLLM FP8 KV cacheOptionally, attention layer types that are more sensitive, such as sliding-window layers[36]
Transformer Engine NVFP4 trainingThe last few layers, recommended in higher precision such as MXFP8[7]
Meta GEM MXFP8 trainingLater, more precision-sensitive layers in BF16[24]

Expanded article table

Meta's Llama 3 paper gives a detailed account of why such exceptions matter. The team quantized the feed-forward layers, which account for roughly 50 percent of inference compute time, used row-wise rather than tensor-wise scales, and capped dynamic scale factors at 1,200 because high-perplexity tokens such as dates produced large activations and underflows. It reported that standard benchmarks often show FP8 inference on par with BF16 even without these mitigations, yet without the cap the model occasionally produced corrupted responses, so it compared reward-model score distributions over 100,000 FP8 and BF16 responses instead [33].

The LM head, the final projection from hidden states to vocabulary logits, is a common exception: the quantization examples above leave it in 16-bit precision, and checkpoints often ship it unquantized. During single-stream decoding, however, its whole weight matrix is read for every generated token, which makes it a target when decoding is limited by memory bandwidth [35].

Example: online FP8 LM head on DGX Spark

An October 27, 2026 NVIDIA technical blog post on Ling-3.0-flash, an Inclusion AI mixture-of-experts model with 124 billion total and 5.1 billion active parameters, measured this trade-off on NVIDIA DGX Spark [35]. Profiling at batch size 1 showed that decoding was memory-bound, with the MoE computation and the output-layer projection standing out. The FP4 checkpoint stores the LM head, a 2,560-to-157,184 vocabulary projection, in BF16 and lists it in modules_to_not_convert, and the cookbook baseline additionally used an FP32 LM-head option. NVIDIA's optimization applies dynamic FP8 quantization only when constructing the LM head, while the rest of the model loads with the checkpoint's FP8 and MXFP4 configuration. Moving the LM-head weights from BF16 to FP8 halves that layer's read volume. The setting is enabled with the environment variable SGLANG_ENABLE_FP8_LM_HEAD=1 and is mutually exclusive with the --enable-fp32-lm-head option [35].

Configuration (synthetic 32K input, 2K output, batch size 1)Time per output tokenDecode throughputChange vs. baseline
Baseline (FlashInfer MXFP4 MoE kernels, FP32 LM head)27.71 ms36.09 tokens/sn/a
Plus Humming MoE backend25.35 ms39.45 tokens/s+9.3%
Plus online FP8 LM head22.76 ms43.94 tokens/s+21.7%

Expanded article table

NVIDIA reports that the FP8 LM head added 11.4 percent decode throughput over the Humming backend alone [35]. The post also documents a failure mode. When multi-token prediction speculative decoding is enabled as well, the draft layer must share the entire LM-head module with the target model, not only its weight tensor. Otherwise the framework silently falls back: the draft receives FP8 weights but interprets them as BF16, its accept length drops to 1.00, and throughput falls below that of a configuration with no optimizations [35]. In NVIDIA's full GSM8K run with all three optimizations (Humming, the FP8 LM head, and multi-token prediction) enabled together, accuracy was 96.82 percent against 96.89 percent for the baseline, a change NVIDIA describes as not statistically detectable [35].

How is FP8 used for inference and quantization?

FP8 is also a widely used post-training quantization format. In the common "W8A8" scheme, weights and activations are cast to E4M3, the encoding the 2022 paper recommends for both, cutting weight memory roughly in half relative to 16-bit formats and engaging the hardware's FP8 matrix paths [1][15]. The vLLM documentation cites a 2x reduction in model memory and up to 1.6x higher throughput with minimal accuracy impact, and runs FP8 checkpoints on older Turing and Ampere GPUs as weight-only W8A16 using Marlin kernels [15].

Weight scales are computed once, offline, typically per tensor or per output channel. Activation scales are either static, calibrated offline by running sample data through the model, or dynamic, computed at runtime per tensor or per token. The 2022 paper's post-training experiments used max calibration for weights and chose the best of max, percentile, and MSE calibration for activations [1]. vLLM's recommended FP8_DYNAMIC scheme in LLM Compressor combines static per-channel weight scales with dynamic per-token activation scales and needs no calibration data; vLLM can also quantize a BF16 or FP16 model on the fly with per-tensor weight scales and dynamic per-tensor activation scales, though its documentation notes that latency improvements are limited in that mode [15]. Tools such as NVIDIA's TensorRT Model Optimizer, the vLLM-associated LLM Compressor, and AMD's Quark produce FP8 checkpoints, and engines including vLLM, TensorRT-LLM, and SGLang execute them [15].

The 2022 paper compared FP8 and INT8 post-training quantization of 16-bit-trained models, both with per-channel weight scales and per-tensor activation scales [1]:

ModelMetric16-bitINT8E4M3
BERT BaseSQuAD v1.1 F1 (higher is better)88.1976.8988.09
BERT LargeSQuAD v1.1 F190.8789.6590.94
GPT-3 126Mwikitext103 perplexity (lower is better)19.0128.3719.43
GPT-3 1.3Bwikitext103 perplexity10.1912.7410.29
GPT-3 6.7Bwikitext103 perplexity8.5110.298.41

Expanded article table

The KV cache can also be held in FP8 to fit more tokens in memory. vLLM supports either one scale per tensor or one scale per attention head (the latter calibrated with LLM Compressor), accepts fp8, fp8_e4m3, and fp8_e5m2 cache types, and falls back to scales of 1.0 when no calibration is done; with the FlashAttention 3 backend, attention itself runs in FP8 and queries are quantized as well [36]. FlashAttention-3 introduced FP8 attention on the H100 using block quantization and "incoherent processing", reaching close to 1.2 petaFLOPS; its authors report 2.6 times lower numerical error than a baseline FP8 attention implementation [37].

Benchmark scores alone can overstate how safe a quantized model is. As noted above, Meta found that FP8 inference that looked on par with BF16 on standard benchmarks still produced occasional corrupted responses until it bounded the dynamic scale factors [33].

Native FP8 releases have become routine. Meta published an official FP8-quantized build of Llama 3.1 405B and said the quantization allowed the model to run within a single server node [34]; the Llama 3 paper notes that in BF16 the model does not fit in the memory of one machine with eight H100 GPUs [33]. The FP8 repository holds about 324.5 billion parameters in E4M3 and about 81.3 billion in BF16 [16]. DeepSeek distributes both V3 and R1 in FP8, with E4M3 weights and 128x128 block scales [38][44].

What are microscaling (MX) formats?

A single scale per tensor must compromise between outliers and typical values. Microscaling (MX) formats address this by attaching a shared scale to every small block of elements. The MX Alliance, formed by AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm, published version 1.0 of the OCP Microscaling Formats specification in September 2023 as an open, license-free standard [17][18]. Each MX block holds 32 elements plus one shared 8-bit power-of-two scale (an exponent-only E8M0 value):

FormatElement encoding(s)Element bitsBlock sizeEffective bits per element
MXFP8FP8 E4M3 or E5M28328.25
MXFP6FP6 E3M2 or E2M36326.25
MXFP4FP4 E2M14324.25
MXINT8INT88328.25

Expanded article table

The E8M0 scale is an unsigned exponent with bias 127 covering 2^-127 to 2^127; it has no zero and no infinity, and its single NaN encoding (all ones) makes every value in the block NaN [17]. A companion paper demonstrated direct-cast inference and training with MX formats, including language-model training in MXFP8 and MXFP6 that closely tracked FP32 baselines [19]. NVIDIA's Blackwell tensor cores natively support MXFP8, MXFP6, and MXFP4 alongside NVIDIA's own NVFP4, which differs from MXFP4 by using 16-element blocks with an FP8 (E4M3) scale plus a per-tensor FP32 scale [9]; AMD's MI350 series supports the MX family as well [40]. A 2025 NVIDIA study trained an 8-billion-parameter model on 15 trillion tokens in MXFP8 with accuracy matching BF16, recommending E4M3 elements for every tensor, gradients included [20]. The same study found that the rounding of the E8M0 scale matters: the conversion suggested in the OCP v1.0 specification effectively rounds the scale down, which lets scaled values overflow, while rounding it up toward positive infinity keeps every scaled value within range; the authors report that Blackwell has hardware support for their rounding method [20]. On the deployment side, OpenAI's gpt-oss models (August 2025) ship with mixture-of-experts weights quantized to MXFP4, about 4.25 bits per parameter, which lets the 120-billion-parameter model run on a single 80 GB GPU [21].

MXFP8 training has also reached production recommender systems. Meta said in an August 3, 2026 engineering post that GEM, its Generative Ads Recommendation Model and the foundation model behind ads recommendations across Instagram and Facebook, trains at LLM scale on several thousand latest-generation GPUs with a mixed ultra-low-precision recipe that includes MXFP8 attention and MLP layers, part of a co-design effort the company says doubled end-to-end training efficiency to 20 to 25 percent model FLOPs utilization [24]. Because ads click-through and conversion prediction is highly sensitive to numerical change, Meta extended the FlashAttention 4 kernel with end-to-end MXFP8 block-scaled matrix multiplication for both forward and backward passes, applied Random Hadamard Transforms to spread activation outliers before quantization, and fell back to BF16 in later, more precision-sensitive layers; the company reports over 1.3x forward and over 1.5x backward attention-kernel speedups from MXFP8 on its internal GPUs [24].

How does FP8 compare to FP16, INT8, and FP4?

Versus FP16 and BF16. FP8 halves storage and, on most accelerators, doubles peak tensor throughput relative to 16-bit formats; Intel's Gaudi 3, which is rated at the same 1,835 teraFLOPS for FP8 and BF16, is an exception [39]. E5M2 shares FP16's exponent range, while BF16's 8-bit exponent gives it a far wider range than either FP8 encoding, which is why higher-precision formats are retained for master weights, sensitive operations, and loss computation even in FP8 training runs.

Versus INT8. Both occupy one byte. INT8's uniformly spaced values resolve a well-bounded distribution more finely, whereas FP8's exponent allocates precision logarithmically and tolerates the long-tailed, outlier-heavy activation distributions typical of Transformers. Qualcomm researchers argued in 2023 that FP8 multiply-accumulate units are 50 to 180 percent less area- and energy-efficient than INT8 and that INT8 remains preferable for many convolutional and edge workloads, while finding that E4M3-style formats suit Transformer layers with large outliers [22]. On many server accelerators FP8 and INT8 peak rates are identical (the MI300X lists 2,614.9 for both, in teraFLOPS and tera-operations per second [10]), and FP8's robustness to outliers without elaborate calibration has made it the more common choice for LLM serving. NVIDIA's HGX specifications list the same 72 petaFLOPS of sparse FP8/FP6 tensor throughput for eight-GPU HGX B200 and HGX B300 systems, but INT8 tensor throughput of 72 petaOPS for the B200 system against 3 petaOPS for the Blackwell Ultra-based B300 [41].

Versus FP4. FP4 (E2M1) continues the same trajectory: Blackwell is NVIDIA's first generation with FP4 tensor cores, doubling throughput again over FP8. Block-scaled FP4 (MXFP4 and NVFP4) is today primarily an inference and weight-storage format, with FP4 pretraining an active research area; an NVIDIA-led 2025 paper reported training a 12-billion-parameter model on 10 trillion tokens in NVFP4 [23]. FP8 in turn serves as a building block of FP4 recipes, providing the per-block scale datatype in NVFP4 and the higher-precision fallback for sensitive layers [9].

Continue reading

More to read

Topics:AI ComputeAI HardwareAI InferenceTraining & Optimization

See also

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18Micikevicius, P. et al. "FP8 Formats for Deep Learning". arXiv:2209.05433, September 2022. arxiv.org/...2209.05433
  2. ^1 ^2"NVIDIA, Arm, and Intel Publish FP8 Specification for Standardization as an Interchange Format for AI". NVIDIA Technical Blog, September 2022. developer.nvidia.com/...-interchange-format-for-ai
  3. ^Wang, N. et al. "Training Deep Neural Networks with 8-bit Floating Point Numbers". arXiv:1812.08011 (NeurIPS 2018). arxiv.org/...1812.08011
  4. ^1 ^2 ^3"FP8 Numbers". AMD ROCm HIP documentation. rocm.docs.amd.com/...fp8_numbers
  5. ^1 ^2 ^3"OCP 8-bit Floating Point Specification (OFP8), Revision 1.0". Open Compute Project, 2023. opencompute.org/...8-revision-1-0-2023-12-01-pdf-1
  6. ^"NVIDIA Hopper Architecture In-Depth". NVIDIA Technical Blog, March 2022. developer.nvidia.com/...pper-architecture-in-depth
  7. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15"Using FP8 and FP4 with Transformer Engine". NVIDIA Transformer Engine documentation. docs.nvidia.com/...fp8_primer
  8. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9"Per-Tensor and Per-Block Scaling Strategies for Effective FP8 Training". NVIDIA Technical Blog. developer.nvidia.com/...for-effective-fp8-training
  9. ^1 ^2 ^3"Introducing NVFP4 for Efficient and Accurate Low-Precision Inference". NVIDIA Technical Blog, 2025. developer.nvidia.com/...te-low-precision-inference
  10. ^1 ^2 ^3"AMD Instinct MI300X Accelerator Data Sheet". AMD, 2023. amd.com/...amd-instinct-mi300x-data-sheet.pdf
  11. ^"Intel Gaudi AI Accelerator Gains 2x Performance Leap on GPT-3 with FP8 Software". Intel Newsroom, November 2023. newsroom.intel.com/...tor-brings-greater-ai-choice
  12. ^"Inflection AI debuts new flagship Inflection-2 LLM trained on 5,000 H100 chips". SiliconANGLE, November 22, 2023. siliconangle.com/...-2-llm-trained-5000-h100-chips
  13. ^Peng, H. et al. "FP8-LM: Training FP8 Large Language Models". arXiv:2310.18313, October 2023. arxiv.org/...2310.18313
  14. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10DeepSeek-AI. "DeepSeek-V3 Technical Report". arXiv:2412.19437, December 2024. arxiv.org/...2412.19437
  15. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8"FP8 W8A8". vLLM documentation. docs.vllm.ai/...fp8
  16. ^"Llama-3.1-405B-Instruct-FP8". Meta on Hugging Face, July 2024. huggingface.co/...Llama-3.1-405B-Instruct-FP8
  17. ^1 ^2 ^3"OCP Microscaling Formats (MX) Specification, Version 1.0". Open Compute Project, September 2023. opencompute.org/...-formats-mx-v1-0-spec-final-pdf
  18. ^"AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Standardize Next-Generation Narrow Precision Data Formats for AI". Open Compute Project blog, October 2023. opencompute.org/...w-precision-data-formats-for-ai
  19. ^Rouhani, B. D. et al. "Microscaling Data Formats for Deep Learning". arXiv:2310.10537, October 2023. arxiv.org/...2310.10537
  20. ^1 ^2"Recipes for Pre-training LLMs with MXFP8". arXiv:2506.08027, 2025. arxiv.org/...2506.08027
  21. ^"Introducing gpt-oss". OpenAI, August 5, 2025. openai.com/...introducing-gpt-oss
  22. ^van Baalen, M. et al. "FP8 versus INT8 for efficient deep learning inference". arXiv:2303.17951, March 2023. arxiv.org/...2303.17951
  23. ^"Pretraining Large Language Models with NVFP4". arXiv:2509.25149, September 2025. arxiv.org/...2509.25149
  24. ^1 ^2 ^3Liu, D. et al. "GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model". Meta Engineering Blog, August 3, 2026. engineering.fb.com/...ommendation-foundation-model
  25. ^AI Wiki. "FP8: E4M3 and E5M2 bit fields". CC0 1.0. Original geometric illustration; not a reproduction of a source figure. Technical basis: Micikevicius et al., FP8 Formats for Deep Learning.
  26. ^Sun, X. et al. "Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks". NeurIPS 2019. papers.nips.cc/...97a89789352e211ca2d398f-Abstract
  27. ^"Tesla Dojo Technology: A Guide to Tesla's Configurable Floating Point Formats & Arithmetic". Tesla. digitalassets.tesla.com/...tesla-dojo-technology.pdf
  28. ^Noune, B. et al. "8-bit Numerical Formats for Deep Neural Networks". arXiv:2206.02915, June 2022. arxiv.org/...2206.02915
  29. ^"Graphcore and AMD propose FP8 AI standard with Qualcomm support". Graphcore, July 5, 2022. graphcore.ai/...-ai-standard-with-qualcomm-support
  30. ^Fitzgibbon, A., Wintersteiger, C. M., and Sarnoff, J. "Novel Aspects of IEEE SA P3109 Arithmetic Formats for Machine Learning". arXiv:2606.04028, June 2026. arxiv.org/...2606.04028
  31. ^1 ^2 ^3 ^4"Float stored in 8 bits". ONNX documentation. onnx.ai/...float8
  32. ^1 ^2 ^3"Low precision floating point types". AMD HIP documentation. rocm.docs.amd.com/...low_fp_types
  33. ^1 ^2 ^3 ^4 ^5Grattafiori, A. et al. "The Llama 3 Herd of Models". arXiv:2407.21783, July 2024. arxiv.org/...2407.21783
  34. ^"Introducing Llama 3.1: Our most capable models to date". Meta AI, July 23, 2024. ai.meta.com/...meta-llama-3-1
  35. ^1 ^2 ^3 ^4 ^5 ^6Liu, Y. "3 Ways to Boost MoE Inference on NVIDIA DGX Spark". NVIDIA Technical Blog, October 27, 2026. developer.nvidia.com/...erence-on-nvidia-dgx-spark
  36. ^1 ^2"Quantized KV Cache". vLLM documentation. docs.vllm.ai/...quantized_kvcache
  37. ^Shah, J. et al. "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision". arXiv:2407.08608, July 2024. arxiv.org/...2407.08608
  38. ^1 ^2 ^3"DeepSeek-V3 config.json". DeepSeek on Hugging Face. huggingface.co/...config.json
  39. ^1 ^2 ^3"Intel Gaudi's third, final hurrah posited as H100 contender". The Register, April 9, 2024. theregister.com/...intel_gaudi_ai_accelerator
  40. ^1 ^2 ^3 ^4"AMD Instinct MI355X GPUs". AMD. amd.com/...mi355x
  41. ^1 ^2 ^3 ^4"NVIDIA HGX Platform". NVIDIA. nvidia.com/...hgx
  42. ^1 ^2"Ironwood: The first Google TPU for the age of inference". Google, April 9, 2025. blog.google/...ironwood-tpu-age-of-inference
  43. ^"AWS Trainium". Amazon Web Services. aws.amazon.com/...trainium
  44. ^"DeepSeek-R1 config.json". DeepSeek on Hugging Face. huggingface.co/...config.json
  45. ^1 ^2"NVIDIA H100 Tensor Core GPU" (product specifications). NVIDIA. Accessed October 30, 2026. nvidia.com/...h100

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

7 revisions · v7 · 5,682 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Format definitions checked against the FP8 Formats paper and the OCP 8-bit floating point specification; hardware throughput figures against NVIDIA, AMD, Intel, Google and AWS product pages; training and inference recipes against DeepSeek, Meta, vLLM and Transformer Engine documentation, October 2026.

Cite this page: AI Wiki. "FP8." aiwiki.ai, updated 29 Oct 2026, fact-checked 29 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/fp8

Edit article