# Floating-point arithmetic

> Source: https://aiwiki.ai/wiki/floating_point
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: AI Compute, AI Hardware, Computer Science, Deep Learning
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Floating-point arithmetic." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/floating_point
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Floating-point arithmetic** is the way computers represent and calculate with real numbers using a fixed number of bits split between a sign, an exponent and a significand (also called the mantissa), so that one format can hold both very large and very small values.[1][2] Almost all general-purpose hardware follows the IEEE 754 standard, first published in 1985 and most recently revised in 2019, which defines formats, operations, rounding, special values and exception handling.[2][3][6] Because only finitely many values can be stored, most real numbers (including the decimal 0.1) are rounded, and results can depend on the order in which operations are carried out.[2][4]

In [machine learning](https://aiwiki.ai/wiki/machine_learning), floating-point formats are a central design choice. The arithmetic in [neural networks](https://aiwiki.ai/wiki/neural_network) consists mostly of [dot products](https://aiwiki.ai/wiki/dot_product) (the building block of [matrix multiplication](https://aiwiki.ai/wiki/matrix_multiplication)), reductions and point-wise operations,[8] and machine learning methods reach the accuracy they need at 32-bit precision and below, while simulation codes mostly require 64-bit accuracy.[21] Since the late 2010s, AI hardware and software have moved from 32-bit single precision (FP32) to 16-bit formats such as FP16 and [bfloat16](https://aiwiki.ai/wiki/bfloat16), then to 8-bit [FP8](https://aiwiki.ai/wiki/fp8) and 4-bit [FP4](https://aiwiki.ai/wiki/fp4), usually inside a [mixed-precision](https://aiwiki.ai/wiki/mixed_precision_training) scheme that keeps sensitive values in higher precision.[8][9][15][18] High-performance scientific computing still measures itself mainly in 64-bit double precision (FP64).[20][21]

<figure>
<img src="/images/media-2026-11-18-floating_point-12ac8cffcadb.png" alt="A 32-bit row of bits split into a 1-bit sign, an 8-bit exponent and a 23-bit fraction, equal to 0.15625" width="1200" height="153" />
<figcaption>Bit layout of a 32-bit single-precision floating-point number, here encoding 0.15625, with its sign, 8-bit exponent and 23-bit fraction fields. Vectorization: Stannered, via <a href="https://commons.wikimedia.org/wiki/File:Float_example.svg">Wikimedia Commons</a>, <a href="http://creativecommons.org/licenses/by-sa/3.0/">CC BY-SA 3.0</a>.</figcaption>
</figure>

## Representation

A binary floating-point number has three fields: a one-bit sign, an exponent stored with a fixed offset called the bias, and the fraction bits of the significand.[1][2] For an ordinary (normal) number, the value is

$$x = (-1)^{s} \times 1.f \times 2^{\,e - \text{bias}}$$

where $s$ is the sign bit, $f$ is the stored fraction read as a binary fraction, and $e$ is the stored exponent field. The leading 1 of the significand is implied and not stored, so the 23 stored fraction bits of single precision give 24 bits of precision and the 52 stored bits of double precision give 53.[1] Single precision uses an 8-bit exponent with a bias of 127; double precision uses an 11-bit exponent with a bias of 1023.[1][2] Microsoft's documentation gives the example of 0.75, which is stored as $1.5 \times 2^{-1}$ with a biased exponent of 126 (hexadecimal `0x3F400000`).[1]

Some exponent bit patterns are reserved for special values:[1]

| Encoding | Exponent field | Fraction field | Meaning |
|---|---|---|---|
| Zero | all zeros | all zeros | +0 or -0 (compare as equal) |
| Subnormal (denormal) | all zeros | non-zero | Tiny values with an implied leading 0 instead of 1 |
| Infinity | all ones | all zeros | +∞ or -∞ |
| NaN (Not a Number) | all ones | non-zero | Undefined results such as 0/0 |

Subnormal numbers fill the gap between zero and the smallest normal number, at the cost of fewer significant bits as they get smaller.[1] Their inclusion, known as gradual underflow, was a major point of contention during the drafting of IEEE 754.[5] Quiet NaNs propagate through the rest of an expression.[1] In training, overflows produce infinities and NaNs in the weight gradients, which irreversibly damage the weights once an update is applied (see [NaN trap](https://aiwiki.ai/wiki/nan_trap)).[8]

### Representation error

Most decimal fractions have no exact binary form. The Python documentation explains that 1/10 in base 2 is the infinitely repeating fraction 0.000110011001100..., so a double-precision float stores the nearest value it can, which is 3602879701896397 / 2^55.[4] The stored number printed in full is 0.1000000000000000055511151231257827021181583404541015625, and as a result `0.1 + 0.1 + 0.1 == 0.3` evaluates to `False` in Python.[4]

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/PZRI1IfStY0" title="Floating Point Numbers - Computerphile"></iframe>
<figcaption>"Floating Point Numbers - Computerphile", video from Computerphile, published January 22, 2014. <a href="https://www.youtube.com/watch?v=PZRI1IfStY0">Watch on YouTube</a>.</figcaption>
</figure>

## The IEEE 754 standard

### Origins

In the 1960s and 1970s, each line of computers had its own floating-point range, precision and rounding behavior, and programmers had to work around anomalies such as numbers that acted as non-zero in comparisons but as zero in multiplication.[5] According to William Kahan's account, Intel began designing a floating-point coprocessor for its 8086/8088 and i432 processors in 1976, and Intel's John Palmer recruited Kahan as a consultant on the arithmetic for what became the 8087 coprocessor.[5]

A separate effort under the IEEE, the p754 working group, held its second meeting in November 1977 under the chairmanship of Richard Delp, with attendees from microprocessor makers including National Semiconductor, Zilog and Motorola.[5] With Intel's permission to disclose the 8087's formats and special values, Kahan, his University of California, Berkeley student Jerome Coonen and visiting professor Harold Stone wrote a draft specification, known as the "K-C-S" draft, which p754 eventually adopted.[5] The biggest dispute was over gradual underflow, which opponents feared would slow down fast hardwired floating-point units.[5] By 1984, Kahan recalled, the draft had already been implemented by Intel, AMD, Apple, IBM, Motorola, National Semiconductor, Weitek, Zilog and AT&T, among others, and he said that "a year before it was canonized, IEEE 754-1985 had become a de facto standard".[5] Kahan received the ACM Turing Award in 1989.[5]

### Revisions

| Version | Title | Board approval | Published | Notes |
|---|---|---|---|---|
| IEEE 754-1985 | IEEE Standard for Binary Floating-Point Arithmetic | 1985-03-21 | 1985-10-12 | Basic and extended formats, arithmetic, conversions, exception handling[6] |
| IEEE 754-2008 | IEEE Standard for Floating-Point Arithmetic | 2008-06-12 | 2008-08-29 | Superseded 754-1985; added fused multiply-add[7][2] |
| IEEE 754-2019 | IEEE Standard for Floating-Point Arithmetic | 2019-06-13 | 2019-07-22 | Superseded 754-2008; binary and decimal arithmetic[3] |

The 2019 standard specifies interchange and arithmetic formats and methods for binary and decimal floating-point arithmetic, along with exception conditions and their default handling, and permits implementation in hardware, software or both.[3] NVIDIA's floating-point white paper states that since the 1985 standard was adopted, virtually all mainstream computing systems have implemented it.[2]

A separate IEEE working group, P3109, is drafting a standard for arithmetic formats aimed at machine learning. According to a June 2026 paper describing the draft, it defines a parameterized family of binary floating-point formats that vary in width, precision, signedness and whether infinities are present, makes its operations exception-free (reporting exceptional cases through return values such as NaN), includes stochastic rounding among its rounding modes, and defines operations on blocks of values that share a scale factor.[24]

### Rounding and fused multiply-add

IEEE 754 requires a set of operations including addition, subtraction, multiplication, division, square root, fused multiply-add, remainder, conversions, scaling, sign operations and comparisons, and their results are guaranteed to be the same on every conforming implementation for a given format and rounding mode.[2] The most frequently used rounding mode is round-to-nearest, ties-to-even.[2] The fused multiply-add (FMA) operation, added in the 2008 revision, computes $x \times y + z$ with a single rounding step instead of two, which makes the result more accurate.[2] FMA is built into the hardware of NVIDIA GPUs.[2]

## Precision, range and error

Each format trades range against precision. The exponent width sets the dynamic range, and the fraction width sets how finely values are spaced; Google's description of bfloat16, for example, equates dynamic range with the number of exponent bits.[11] For a format with $m$ fraction bits, the gap between 1 and the next larger representable number is $2^{-m}$, and for an IEEE-style format whose top exponent code is reserved for infinities and NaNs, the largest finite value is $(2 - 2^{-m}) \times 2^{\text{bias}}$. Plugging in the IEEE parameters gives a spacing of about $1.2 \times 10^{-7}$ and a maximum of about $3.4 \times 10^{38}$ for FP32, and about $2.2 \times 10^{-16}$ and $1.8 \times 10^{308}$ for FP64; Microsoft's documentation describes $2^{127}$, roughly $10^{38}$, as the practical limit of single precision.[1]

Because every operation rounds, floating-point arithmetic is not associative: $(a + b) + c$ and $a + (b + c)$ can give different answers. NVIDIA's white paper works through a single-precision example in which the two orderings round to different results, even though each individual operation complies with IEEE 754.[2] The same paper shows that the choice of algorithm for a short dot product changes the accuracy of the final result.[2]

A second effect is that adding a small number to a much larger one can lose the small number entirely. In the 2017 mixed-precision training paper, the authors noted that in FP16, a weight update becomes zero when the weight is more than 2048 times larger than the update, because aligning the binary points shifts the smaller value out of the 10 available fraction bits.[8] They also noted that any value smaller in magnitude than $2^{-24}$ becomes zero in FP16, and that the largest representable FP16 value is 65,504.[8]

## Formats used in AI computing

The following table lists the floating-point formats most often used for deep learning. Fraction bits exclude the implied leading bit; the "spacing near 1" column is $2^{-m}$.

| Format | Other names | Sign / exponent / fraction bits | Largest finite value | Spacing near 1 | Defined by |
|---|---|---|---|---|---|
| FP64 | binary64, double precision | 1 / 11 / 52 | about $1.8 \times 10^{308}$ | $2^{-52}$ | IEEE 754[1] |
| FP32 | binary32, single precision | 1 / 8 / 23 | about $3.4 \times 10^{38}$ | $2^{-23}$ | IEEE 754[1] |
| TF32 | TensorFloat-32 | 1 / 8 / 10 | same range as FP32 | $2^{-10}$ | NVIDIA (Ampere)[12] |
| BF16 | bfloat16, Brain floating point | 1 / 8 / 7 | same range as FP32 | $2^{-7}$ | Google Brain[10][11] |
| FP16 | binary16, half precision | 1 / 5 / 10 | 65,504 | $2^{-10}$ | IEEE 754[8][9] |
| FP8 E5M2 | | 1 / 5 / 2 | 57,344 | $2^{-2}$ | NVIDIA, Arm, Intel (2022)[9] |
| FP8 E4M3 | | 1 / 4 / 3 | 448 | $2^{-3}$ | NVIDIA, Arm, Intel (2022)[9] |
| FP6 E3M2 | | 1 / 3 / 2 | 28 | $2^{-2}$ | OCP MX specification[17] |
| FP6 E2M3 | | 1 / 2 / 3 | 7.5 | $2^{-3}$ | OCP MX specification[17] |
| FP4 E2M1 | | 1 / 2 / 1 | 6 | $2^{-1}$ | OCP MX specification[17] |

### FP64 and FP32

Double precision (FP64) corresponds to the C `double` type, and almost all platforms map Python floats to it.[1][4] It remains the reference precision for high-performance computing. The High-Performance Linpack (HPL) benchmark, long used to benchmark the largest supercomputers, solves a dense linear system in 64-bit double-precision arithmetic.[20][21] The companion HPL-MxP benchmark is meant to highlight the convergence of simulation and AI workloads, noting that simulation codes mostly need 64-bit accuracy while machine learning works at 32 bits and below; it performs the LU factorization in low precision (likely 16-bit) and then uses iterative refinement to recover a 64-bit-accurate solution.[21]

Single precision (FP32), the C `float` type, was the standard format for deep learning training before mixed precision became common: the 2017 mixed-precision paper described FP32 as the format used by modern deep learning training systems at the time.[1][8]

### TF32

TensorFloat-32 (TF32) is a compute mode introduced in the Tensor Cores of NVIDIA's [Ampere](https://aiwiki.ai/wiki/nvidia_ampere) architecture. It has 1 sign bit, 8 exponent bits and 10 mantissa bits, so it covers the same range as FP32 with the same precision as FP16.[12] TF32 is not a storage type: Tensor Cores round FP32 inputs to TF32, multiply them, and accumulate the products in FP32, while tensors in memory stay in FP32.[12] NVIDIA listed the relative math throughput of the [A100](https://aiwiki.ai/wiki/nvidia_a100) as 1x for FP32, 8x for TF32 and 16x for FP16 or BF16.[12]

TF32 became the default for A100 in NVIDIA's framework containers starting with version 20.06, and was enabled by default in [PyTorch](https://aiwiki.ai/wiki/pytorch) 1.7 and [TensorFlow](https://aiwiki.ai/wiki/tensorflow) 2.4.[12] PyTorch later reversed the default for matrix multiplications: its `allow_tf32` flag for matmuls defaulted to true from version 1.7 to 1.11 and to false from 1.12 onward, while the cuDNN convolution flag stayed on.[13] In a PyTorch documentation example on an A100, a large matrix multiplication ran about 7 times faster with TF32 enabled, with a relative error about two orders of magnitude larger than in FP32 when both were compared with a double-precision result.[13] The current documentation recommends newer `fp32_precision` settings and warns that the old `allow_tf32` flags are going to be deprecated.[13]

### FP16

FP16 is the IEEE 754 half-precision format. Its 5-bit exponent gives normal values from $2^{-14}$ up to 65,504, a much narrower range than FP32.[8][9] The 2017 paper by researchers at Baidu and NVIDIA showed that networks could be trained with weights, activations and gradients stored in FP16 without losing accuracy, provided three safeguards were used: an FP32 master copy of the weights, loss scaling, and FP16 arithmetic that accumulates into FP32.[8] Loss scaling multiplies the loss before [backpropagation](https://aiwiki.ai/wiki/backpropagation) so that small gradients move into the representable range, then divides the weight gradients by the same factor before the update; the authors used scaling factors from 8 to 32K and noted that overflows could be detected and the update skipped.[8]

### bfloat16

bfloat16 keeps FP32's 8-bit exponent and shortens the mantissa to 7 stored bits (8 bits of precision), so an FP32 value can be converted by rounding away its lower 16 bits.[10] The name comes from Google Brain, where the format was conceived, and Google's Cloud TPUs perform each multiplication in bfloat16 and accumulate in 32-bit IEEE floating point.[11] Intel and Facebook researchers reported in 2019 that bfloat16 was first introduced in the DistBelief and TensorFlow frameworks as a storage format to reduce communication between machines.[10] Their study found that training with bfloat16 tensors matched FP32 results across image, speech, language, generative and recommendation workloads with no change to hyperparameters, and attributed the appeal of the format to two properties: it has the same range as FP32, and conversion to and from FP32 is simple.[10]

The hardware argument is that multiplier area grows with the square of the mantissa width. Google said bfloat16 multipliers are about half the silicon size of a typical FP16 multiplier and eight times smaller than an FP32 multiplier.[11]

### FP8

FP8 was proposed in September 2022 by researchers from NVIDIA, Arm and Intel as two encodings: E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2 mantissa bits).[9] E5M2 follows IEEE 754 conventions for infinities and NaNs and can be read as FP16 with fewer mantissa bits. E4M3 drops infinities and keeps a single NaN mantissa pattern, which raises its maximum value from 240 to 448.[9] The authors recommended E4M3 for weights and activations and E5M2 for gradients, and reported that FP8 training matched 16-bit results on CNNs, RNNs and Transformer models up to 175 billion parameters without changing hyperparameters.[9] Because FP8's range is narrow, tensors are multiplied by a scaling factor before conversion so that their largest values land near the format's maximum, and the authors argued against the update-skipping approach used with FP16, since FP8 overflows are much more frequent.[9]

NVIDIA's [Hopper](https://aiwiki.ai/wiki/nvidia_hopper) architecture, described in March 2022, introduced FP8 in its fourth-generation [Tensor Cores](https://aiwiki.ai/wiki/tensor_core), along with a Transformer Engine that chooses between FP8 and 16-bit calculations and handles re-casting and scaling layer by layer.[14]

### FP6, FP4 and block-scaled formats

Below 8 bits, a single scale per tensor is not enough to cover the spread of values, so newer formats attach a shared scale to small blocks of elements.[15] In 2023, AMD, Arm, Intel, Meta, Microsoft, NVIDIA and Qualcomm formed the Microscaling Formats (MX) Alliance and released the MX Specification v1.0 through the [Open Compute Project](https://aiwiki.ai/wiki/open_compute_project) in what the OCP called an "open, license-free format".[16] The specification defines MXFP8, MXFP6, MXFP4 and MXINT8, in which each block of 32 elements shares an 8-bit power-of-two scale.[15][17] Version 1.1, dated 9 October 2026, renamed the E8M0 scale type to UE8M0, refined conversion and rounding rules, clarified NaN and out-of-range behavior, and lists Google among the contributors.[17] The FP6 and FP4 element types have no encodings for infinity or NaN.[17] See [Microscaling formats](https://aiwiki.ai/wiki/microscaling_formats) and [MXFP4](https://aiwiki.ai/wiki/mxfp4) for details.

NVIDIA's [Blackwell](https://aiwiki.ai/wiki/nvidia_blackwell) architecture added another 4-bit format, [NVFP4](https://aiwiki.ai/wiki/nvfp4). It uses the same E2M1 element, whose values run from about -6 to 6, but pairs each block of 16 values with an E4M3 FP8 scale, plus a second FP32 scale for the whole tensor.[18] NVIDIA argued that the fractional E4M3 scale represents a block's values more accurately than a power-of-two scale.[18]

## Mixed precision and accumulation

The common thread in low-precision AI arithmetic is that products are computed in a narrow format but summed in a wider one. The 2017 mixed-precision paper found that some networks matched FP32 accuracy only if FP16 dot products accumulated into FP32, and it recommended doing large reductions, such as those in batch normalization and softmax, in FP32.[8] NVIDIA's Volta GPUs introduced Tensor Cores that multiply FP16 matrices and accumulate into FP16 or FP32.[8] The FP8 specification paper likewise expected FP8 operations to produce higher-precision outputs.[9]

The [DeepSeek-V3](https://aiwiki.ai/wiki/deepseek_v3) technical report showed how these details matter at scale. DeepSeek trained its 671-billion-parameter [mixture-of-experts](https://aiwiki.ai/wiki/mixture_of_experts) model with an FP8 framework that ran the main matrix multiplications in FP8 while keeping the embedding module, output head, gating, normalization and attention in BF16 or FP32, and storing master weights, weight gradients and optimizer states in higher precision.[19] It used E4M3 for all tensors and scaled activations in 1×128 tiles and weights in 128×128 blocks, an approach the report compares with microscaling formats.[19] The team also found that FP8 matrix-multiply accumulation on [NVIDIA H800](https://aiwiki.ai/wiki/nvidia_h800) Tensor Cores kept only about 14 bits, and worked around it by copying partial sums to FP32 registers every 128 elements.[19] In validation runs at two smaller model scales, trained on about 1 trillion tokens, the relative loss error of FP8 training stayed below 0.25% compared with a BF16 baseline.[19]

## Numerical issues in practice

**Underflow and overflow.** Narrow formats push small gradients to zero and large activations to infinity. Loss scaling, per-tensor scaling and block scaling are all ways of moving values into the representable range.[8][9][15]

**Rounding mode.** IBM researchers showed in 2015 that networks could be trained using 16-bit fixed-point numbers if they used stochastic rounding, which rounds up or down at random so that the expected result equals the exact value.[23] Under round-to-nearest, any parameter update smaller than half the format's smallest step is always rounded to zero, whereas stochastic rounding gives such updates a non-zero chance of surviving.[23] The FP8 specification leaves the choice of rounding mode, including stochastic rounding, to implementations.[9]

**Reproducibility.** Non-associativity means the same computation can give slightly different results if numbers are summed in a different order. A September 2025 analysis by Horace He and colleagues at [Thinking Machines Lab](https://aiwiki.ai/wiki/thinking_machines_lab) argued that the main cause of run-to-run differences in [large language model](https://aiwiki.ai/wiki/large_language_model) inference servers is not concurrency on the GPU but a lack of "batch invariance": kernels sum in a different order depending on the batch size, which changes with server load.[22] They proposed batch-invariant kernels as the fix.[22]

**Alternative number systems.** Researchers have proposed other representations for deep learning, including logarithmic formats and posits. The 2022 FP8 paper noted that none had been adopted in practice because their benefits had not justified new arithmetic hardware.[9] The same paper described fixed-point INT8 as a popular option for inference; see [quantization](https://aiwiki.ai/wiki/quantization).[9]

## Timeline

| Date | Event |
|---|---|
| 1976 | Intel begins designing the floating-point coprocessor that became the 8087[5] |
| 1977-11 | Second meeting of the IEEE p754 working group[5] |
| 1985-03-21 | IEEE 754-1985 approved[6] |
| 2008-06-12 | IEEE 754-2008 approved, adding fused multiply-add[7][2] |
| 2017-10 | "Mixed Precision Training" paper (Baidu and NVIDIA) posted to arXiv[8] |
| 2019-06-13 | IEEE 754-2019 approved[3] |
| 2019-08 | Google publishes its account of bfloat16 on Cloud TPUs[11] |
| 2021-01 | NVIDIA technical blog details TF32 Tensor Cores on Ampere[12] |
| 2022-03 | NVIDIA Hopper architecture introduces FP8 Tensor Cores[14] |
| 2022-09 | "FP8 Formats for Deep Learning" (NVIDIA, Arm, Intel) posted to arXiv[9] |
| 2023-10 | OCP announces the MX Specification v1.0[16] |
| 2024-12 | DeepSeek-V3 technical report describes large-scale FP8 training[19] |
| 2025-06 | NVIDIA technical blog introduces the NVFP4 format of its Blackwell GPUs[18] |
| 2026-10-09 | OCP MX Specification v1.1[17] |

## See also

- [bfloat16](https://aiwiki.ai/wiki/bfloat16)
- [FP8](https://aiwiki.ai/wiki/fp8)
- [FP4](https://aiwiki.ai/wiki/fp4)
- [FP16](https://aiwiki.ai/wiki/fp16)
- [Mixed-precision training](https://aiwiki.ai/wiki/mixed_precision_training)
- [Microscaling formats](https://aiwiki.ai/wiki/microscaling_formats)
- [Quantization](https://aiwiki.ai/wiki/quantization)
- [NaN trap](https://aiwiki.ai/wiki/nan_trap)
- [Tensor Core](https://aiwiki.ai/wiki/tensor_core)

## References

1. Microsoft. "IEEE Floating-Point Representation". Microsoft Learn (C++ documentation). https://learn.microsoft.com/en-us/cpp/build/ieee-floating-point-representation
2. Nathan Whitehead and Alex Fit-Florea. "Floating Point and IEEE 754 Compliance for NVIDIA GPUs". NVIDIA CUDA documentation. https://docs.nvidia.com/cuda/floating-point/index.html
3. IEEE Standards Association. "IEEE 754-2019: IEEE Standard for Floating-Point Arithmetic". IEEE, 2019. https://standards.ieee.org/ieee/754/6210/
4. Python Software Foundation. "Floating-Point Arithmetic: Issues and Limitations". The Python Tutorial, Python 3 documentation. https://docs.python.org/3/tutorial/floatingpoint.html
5. Charles Severance. "An Interview with the Old Man of Floating-Point: Reminiscences elicited from William Kahan". University of California, Berkeley, 20 February 1998. https://people.eecs.berkeley.edu/~wkahan/ieee754status/754story.html
6. IEEE Standards Association. "IEEE 754-1985: IEEE Standard for Binary Floating-Point Arithmetic". IEEE, 1985. https://standards.ieee.org/ieee/754/993/
7. IEEE Standards Association. "IEEE 754-2008: IEEE Standard for Floating-Point Arithmetic". IEEE, 2008. https://standards.ieee.org/ieee/754/4211/
8. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu. "Mixed Precision Training". ICLR 2018 (arXiv:1710.03740), 2017. https://arxiv.org/abs/1710.03740
9. Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, Hao Wu. "FP8 Formats for Deep Learning". arXiv:2209.05433, 2022. https://arxiv.org/abs/2209.05433
10. Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, et al. "A Study of BFLOAT16 for Deep Learning Training". arXiv:1905.12322, 2019. https://arxiv.org/abs/1905.12322
11. Shibo Wang and Pankaj Kanwar. "BFloat16: The secret to high performance on Cloud TPUs". Google Cloud Blog, 23 August 2019. https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus
12. Dusan Stosic and Paulius Micikevicius. "Accelerating AI Training with NVIDIA TF32 Tensor Cores". NVIDIA Technical Blog, 27 January 2021. https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/
13. PyTorch. "CUDA semantics". PyTorch 2.15 documentation. https://docs.pytorch.org/docs/2.15/notes/cuda.html
14. Michael Andersch, Greg Palmer, Ronny Krashinsky, Nick Stam, Vishal Mehta, Gonzalo Brito, Sridhar Ramaswamy. "NVIDIA Hopper Architecture In-Depth". NVIDIA Technical Blog, 22 March 2022. https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/
15. Bita Darvish Rouhani, Ritchie Zhao, Ankit More, et al. "Microscaling Data Formats for Deep Learning". arXiv:2310.10537, 2023. https://arxiv.org/abs/2310.10537
16. Open Compute Project Foundation. "AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Standardize Next-Generation Narrow Precision Data Formats for AI". Open Compute Project, 17 October 2023. https://www.opencompute.org/news/amd-arm-intel-meta-microsoft-nvidia-and-qualcomm-standardize-next-generation-narrow-precision-data-formats-for-ai/
17. Open Compute Project. "OCP MicroXcaling (MX) Specification, Version 1.1". Open Compute Project, 9 October 2026. https://www.opencompute.org/wp-content/uploads/2026/10/OCP_MicroXcaling_Spec_v1.1.0.pdf
18. Eduardo Alvarez, Omri Almog, Eric Chung, Simon Layton, Dusan Stosic, Ronny Krashinsky, Kyle Aubrey. "Introducing NVFP4 for Efficient and Accurate Low-Precision Inference". NVIDIA Technical Blog, 24 June 2025. https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/
19. DeepSeek-AI. "DeepSeek-V3 Technical Report". arXiv:2412.19437, 2024. https://arxiv.org/abs/2412.19437
20. A. Petitet, R. C. Whaley, J. Dongarra, A. Cleary. "HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers". Netlib, version 2.3, 2 December 2018. https://www.netlib.org/benchmark/hpl/
21. HPL-MxP. "HPL-MxP Mixed-Precision Benchmark". hpl-mxp.org. https://hpl-mxp.org/
22. Horace He and Thinking Machines Lab. "Defeating Nondeterminism in LLM Inference". Thinking Machines Lab: Connectionism, 10 September 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
23. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, Pritish Narayanan. "Deep Learning with Limited Numerical Precision". arXiv:1502.02551, 2015. https://arxiv.org/abs/1502.02551
24. Andrew Fitzgibbon, Christoph M. Wintersteiger, Jeffrey Sarnoff. "Novel Aspects of IEEE SA P3109 Arithmetic Formats for Machine Learning". arXiv:2606.04028, 2026. https://arxiv.org/abs/2606.04028

