FP16 (half precision)
FP16 (also called half precision, binary16 or float16) is a 16-bit floating-point number format with one sign bit, five exponent bits and ten stored significand bits, specified as the binary16 format of the IEEE 754 standard for floating-point arithmetic.[1][2] It takes half the memory of 32-bit single precision (FP32), and in deep learning it was the 16-bit format of the mixed-precision training method that NVIDIA and Baidu researchers published in 2017, which showed that neural networks trained with FP16 storage and arithmetic, an FP32 master copy of the weights and loss scaling could match FP32 accuracy.[3] Its narrow range (the largest finite value is 65,504) is its main weakness, and later large models such as BLOOM were trained in bfloat16, a 16-bit format that keeps FP32's eight-bit exponent.[4][5] FP16 is still used for inference and in graphics and signal processing.[6][7][9][10]
Format
Bit layout
An FP16 value is a 16-bit word split into three fields.[1]
| Bits | Field | Width | Role |
|---|---|---|---|
| 15 | Sign (s) | 1 bit | 0 for positive, 1 for negative |
| 14-10 | Exponent (e) | 5 bits | Biased exponent, bias 15 |
| 9-0 | Significand (m) | 10 bits | Fraction bits after the binary point |
The stored exponent field $e$ selects how the bits are read:[1]
- If $1 \le e \le 30$, the number is normal: $x = (-1)^s \times 2^{e-15} \times 1.m$, where $1.m$ is the binary fraction with an implicit leading 1.
- If $e = 0$ and $m \neq 0$, the number is subnormal (denormalized): $x = (-1)^s \times 2^{-14} \times 0.m$.
- If $e = 0$ and $m = 0$, the value is zero (positive or negative, depending on the sign bit).
- If $e = 31$, the value is infinity when $m = 0$ and NaN (not a number) otherwise.
With the implicit bit, a normal FP16 number carries 11 bits of significand precision, and normal values have exponents from -14 to 15.[3][11] The Imath library, whose half type was formerly distributed with OpenEXR as part of IlmBase, documents this layout and describes the type as compatible with the IEEE 754-2008 binary16 type.[1][12]
Notable values
The Imath header encodes the most negative finite value, -65,504, as 0xFBFF; clearing the sign bit gives 0x7BFF.[1]
The machine epsilon, the gap between 1 and the next larger FP16 number, is $2^{-10} \approx 0.00097656$, the value Imath defines as HALF_EPSILON, and Imath counts three decimal digits that survive a round trip through the format unchanged.[1] Higham and Pranesh give its unit roundoff as $4.88 \times 10^{-4}$.[2] Every integer up to 2048 in magnitude can be represented exactly; 2049 is the first integer that cannot be, and above 2048 the representable integers thin out quickly.[10] Between $2^{-14}$ and $2^{15}$ FP16 can express about 30,720 distinct values.[13]
Range and its consequences
NVIDIA's mixed-precision guide describes FP16's dynamic range, counting subnormals, as 40 powers of two, against 264 for FP32.[14] Intel's guide points out three practical effects of that small range: subnormal numbers are "not very small", so values quickly fall below what normal FP16 numbers can hold; infinity begins only slightly below the largest unsigned 16-bit integer, so converting 16-bit integers can overflow; and rounding integers above 2048 to FP16 introduces large absolute errors.[10] In deep learning, the narrow range shows up as overflow in large activations and as small gradients rounding to zero.[3][8][14]
History
Graphics origins
Sixteen-bit floating point also has a history in computer graphics. OpenEXR, an image file format developed at Industrial Light & Magic (ILM) and first released as open source in 2003, stores HALF channels with the 1-5-10 layout, supporting normal and subnormal numbers, infinities and NaNs.[9][12] Its documentation states that its half class is identical to NVIDIA's 16-bit floating-point format (fp16/half), so 16-bit data can move between OpenEXR files and NVIDIA 16-bit frame buffers without losing bits.[9] The Imath header names Florian Kainz and Rod Bogart of ILM as the primary original authors of the half type.[1]
Standardization
For about two decades after the 1985 IEEE 754 standard, virtually all floating-point hardware implemented its single (fp32) or double (fp64) precision. The 2008 revision of the standard defined a 16-bit half precision format, but as a storage format: it did not define arithmetic operations on it.[2] Hardware vendors nonetheless implemented half-precision arithmetic by extending the rules used for single and double precision.[2] The current edition, IEEE 754-2019, was approved on 2019-06-13 and superseded the 2008 edition; the IEEE also lists ISO/IEC/IEEE 60559:2020, an international floating-point standard with the same scope, among the 754 working group's active standards.[15] Intel describes its AVX-512 FP16 instructions as compliant with IEEE 754-2019, the edition it calls the current standard.[10]
Hardware support
| Year | Hardware | FP16 capability |
|---|---|---|
| 2016 | NVIDIA Tesla P100 (Pascal) | FP16 arithmetic at twice the FP32 rate, 21.2 TFLOPS on the SXM2 module[13] |
| 2017 | NVIDIA V100 (Volta) | 640 Tensor Cores multiplying FP16 matrices with FP16 or FP32 accumulation, up to 125 Tensor TFLOPS[16] |
| 2017 | AMD Radeon Instinct MI25 | Half-precision arithmetic[2] |
| 2020 | NVIDIA A100 (Ampere) | 312 TFLOPS peak FP16 Tensor Core throughput, with BF16 at the same rate[17] |
The P100 reaches its FP16 peak only with paired half2 instructions, which operate on two FP16 values packed into one 32-bit register; scalar half instructions run at about half of peak.[13] Other Pascal chips (GP102, GP104 and GP106) ran FP16 at a lower rate, similar to their double-precision rate.[13] Higham and Pranesh also list Arm's NEON architecture among the platforms with half-precision arithmetic.[2] Clang reports native _Float16 arithmetic on AArch64 from ARMv8.2-A, on AMD and NVIDIA GPU targets, and on x86 when AVX512-FP16 is available.[11]
On x86 processors, Intel's 3rd generation Core processors and their successors could only convert between half precision and FP32, so numeric work had to be done in a higher precision.[10] Intel then added an AVX-512 FP16 instruction set to its 4th generation Xeon Scalable processors, with general-purpose FP16 arithmetic, fused multiply-add, comparisons, conversions and native complex-number support.[10] Intel's guide to these instructions (first revision October 2021) says FP16 handles twice as many operations per clock cycle as FP32 and four times as many as FP64, and it lists signal processing, media and video processing, artificial intelligence and machine learning as typical uses.[10]
Higham and Pranesh cite one example of what the format offers outside machine learning: the Summit supercomputer at Oak Ridge National Laboratory had a peak of 148.6 petaflops on the double-precision LINPACK benchmark, but reached 2.36 exaflops on a genetics application that used half precision.[2]
Use in deep learning
Mixed precision training
In "Mixed Precision Training", published at ICLR 2018 by Paulius Micikevicius and colleagues from NVIDIA and Baidu Research, weights, activations and gradients are stored in FP16, and three techniques keep accuracy at the FP32 level:[3]
- FP32 master weights. An FP32 copy of the weights receives the updates in the optimizer step, and an FP16 copy is used in the forward and backward passes. The paper gives two reasons. Updates smaller than $2^{-24}$ become zero in FP16; in a Mandarin speech model, about 5% of weight-gradient values had exponents below -24. And an update can vanish when it is added to a weight at least 2048 times larger, because aligning the binary points shifts it out of the 10-bit significand. Updating FP16 weights directly in the speech model caused an 80% relative accuracy loss.[3]
- Loss scaling. The loss is multiplied by a constant before backpropagation, so every gradient is scaled up by the chain rule and fewer of them fall below the FP16 range.[3]
- FP32 accumulation. FP16 dot products accumulate partial products in FP32 and convert back to FP16 when writing to memory, and large reductions, such as batch-normalization statistics and softmax, are computed in FP32.[3]
The extra FP32 weight copy raises weight memory by 50%, but because activations dominate training memory and are stored in FP16, overall memory use is roughly halved.[3] The authors reported matching FP32 results on convolutional and recurrent networks and GANs, including models with more than 100 million parameters.[3] See mixed-precision training for the method in general, including its later BF16 and FP8 forms.
Loss scaling
The scaling problem is concrete. In the Multibox SSD detector, converting activation gradients to FP16 turned 31% of the values into zeros, leaving 5.3% nonzero, and training diverged.[14] Multiplying the gradients by 8 (shifting exponents by three) matched FP32 accuracy, and the paper concluded that activation-gradient values below $2^{-27}$ did not matter for that model while values between $2^{-27}$ and $2^{-24}$ had to be kept.[3][14]
Gradients are divided by the scale factor before the weight update, preferably right after the backward pass and before gradient clipping, so hyperparameters such as the clipping threshold and weight decay need no change.[3] The NVIDIA and Baidu paper and NVIDIA's guide describe two ways of choosing the factor:
| Approach | How it works | Reported settings |
|---|---|---|
| Static (constant) | A fixed factor, chosen by trial or so that the factor times the largest gradient stays below 65,504 | Factors from 8 to 32K across tested networks; many needed none[3] |
| Dynamic | Start large; skip the update and shrink the factor when an overflow (Inf or NaN) appears; grow it after a run of overflow-free steps | NVIDIA's runs used N = 2000 steps, growth factor 2, reduction factor 0.5[14] |
PyTorch's GradScaler implements dynamic scaling with defaults of an initial scale of 65,536, a growth factor of 2, a backoff factor of 0.5 and a growth interval of 2000 steps.[18] The PyTorch documentation explains the need for it directly: small gradient values may not be representable in float16, flush to zero, and the corresponding parameter updates are lost.[18]
Frameworks and automatic mixed precision
NVIDIA released Apex, a PyTorch extension with an automatic mixed precision (AMP) feature, in 2018; for PyTorch 1.6 (2020), developers at NVIDIA and Facebook moved the functionality into PyTorch itself as torch.cuda.amp, and Apex AMP went into maintenance mode.[19] In benchmarks published with that release, AMP with FP16 trained models 1.5 to 5.5 times faster than FP32 on V100 GPUs while reaching the same final accuracy.[19]
In current PyTorch, torch.autocast chooses a precision per operation. On CUDA, operations such as matmul, linear, conv2d and bmm run in float16, while softmax, layer_norm, exp, cross_entropy and sum are promoted to float32.[18] NVIDIA's guide sorts operations the same way: convolutions and fully connected layers run in FP16, large reductions, cross-entropy, L1 loss and exponentials stay in FP32, and element-wise operations follow their inputs.[14]
To use Tensor Cores with FP16 inputs, NVIDIA's guide asks that matrix dimensions (M, N and K) and convolution channel counts be multiples of 8, and recommends padding vocabulary size and sequence length to multiples of 8.[14] The guide quotes overall speedups of up to 3x on the most arithmetically intense models.[14]
Training large language models
Some early large language models were trained with FP16 weights. Meta AI's OPT-175B kept Adam optimizer state in FP32 while the model weights stayed in FP16, and used dynamic loss scaling to avoid underflow; when the loss diverged, the team restarted from points where the dynamic loss scaler was still at or above 1.0, and resetting the scaler helped recover some but not all divergences.[20]
The BLOOM team reported irreversible divergences in earlier 104-billion-parameter experiments on V100 GPUs, which it attributed to float16's limited dynamic range; the final 176B model was trained in bfloat16 mixed precision on A100 GPUs, which the paper says solved the instability problem.[4] The BLOOM paper also notes that float16 has since been held responsible for many of the instabilities observed in LLM training.[4] The PyTorch documentation gives a related warning: most models pretrained in bf16 cannot operate within FP16's maximum of 65,504, so their gradients overflow rather than underflow, and NaNs in the loss or gradients under FP16 mixed precision are a sign that the model may not be compatible.[18] Hugging Face's debugging guide walks through an FP16 overflow in google/mt5-small, where a feed-forward output reached about 6.27e4, near the FP16 maximum, and dropout's rescaling then pushed values to infinity.[8]
Inference
Some inference tools default to FP16. The vLLM inference engine, with its default auto data type, runs FP32 and FP16 checkpoints in FP16 and BF16 checkpoints in BF16.[6] Apple's Core ML converter types an ML program in float16 precision by default. For models in the older neural network format, Apple's documentation says the GPU and Neural Engine run in float16 while the CPU runs in float32; in ML programs the GPU supports both precisions and some operations, such as convolution, can also run in float16 on the CPU.[7] A 2025 preprint by Penghui Qi and colleagues argued that FP16 can also help reinforcement-learning fine-tuning of LLMs: in their account, BF16's large rounding errors break the consistency between the training and inference computations, and switching to FP16 removed that mismatch and gave more stable optimization.[21]
Comparison with bfloat16
bfloat16 (BF16) spends its 16 bits differently: 8 exponent bits and 7 stored significand bits, which gives it FP32's range at much lower precision.[2][5]
| Property | FP16 | BF16 | FP32 |
|---|---|---|---|
| Exponent bits | 5 | 8 | 8 |
| Significand bits (with implicit bit) | 11 | 8 | 24 |
| Largest finite value | 6.55e4 | 3.39e38 | 3.40e38 |
| Smallest normal value | 6.10e-5 | 1.18e-38 | 1.18e-38 |
| Smallest subnormal value | 5.96e-8 | 9.18e-41 | 1.40e-45 |
| Unit roundoff | 4.88e-4 | 3.91e-3 | 5.96e-8 |
Values from Higham and Pranesh's comparison table, rounded to three significant figures.[2] The BF16 subnormal figure applies only where subnormals are supported; Intel's bfloat16 specification flushes them to zero.[2]
The table shows the trade. FP16's unit roundoff is eight times smaller than BF16's, so each FP16 value is more precise, but FP16 covers a far smaller range of magnitudes.[2] Google's Shibo Wang and Pankaj Kanwar wrote in 2019 that FP16 "typically requires special handling via techniques such as loss scaling", while bfloat16 is close to a drop-in replacement for FP32; they also note that a hardware multiplier's area grows with the square of the significand width, which makes a bfloat16 multiplier about half the size of a typical FP16 one.[5] A 2019 study by Dhiraj Kalamkar and colleagues trained image, speech, language, generative and recommendation models in emulated BF16 and reported results comparable to FP32 without changing hyperparameters, in contrast to the tuning FP16 typically needs.[22]
Intel's FP16 guide describes the difference in purpose: bfloat16 is the upper 16 bits of FP32 and suits machine learning, while IEEE FP16 balances precision and range in a way it considers better for general-purpose algorithms.[10] On NVIDIA's A100, BF16 and FP16 Tensor Core operations run at the same rate (312 TFLOPS dense), although outside the Tensor Cores NVIDIA lists peak FP16 at 78 TFLOPS against 39 TFLOPS for BF16.[17] Moving a model between the formats is asymmetric: BF16-trained models can overflow when run in FP16, while every finite FP16 magnitude lies inside BF16's range, although converting to BF16 loses precision.[2][18]
NVIDIA's TensorFloat-32 (TF32) mode on Ampere GPUs combines the two designs, pairing FP32's 8-bit exponent with FP16's 10-bit mantissa.[17]
Programming language and library support
| Environment | FP16 type | Notes |
|---|---|---|
| C (ISO/IEC TS 18661-3:2015) | _Float16 | Interchange floating type; Clang supports it on many targets[11] |
| Clang (Arm extension) | __fp16 | Originally an ARM extension; never used directly for arithmetic, operands promoted to float; Clang uses IEEE binary16 rather than Arm's alternative half-precision format[11] |
| C++23 | std::float16_t in <stdfloat> | Optional; available when __STDCPP_FLOAT16_T__ is defined, literal suffix f16[23] |
| CUDA | half, half2 in cuda_fp16.h | Intrinsics such as __hfma and __hadd2[13] |
| NumPy | numpy.float16 (numpy.half) | Character code 'e'[24] |
| PyTorch | torch.float16 (half) | Used by torch.autocast and GradScaler[18] |
| Imath / OpenEXR | half | C++ class that mixes with float and double; conversion uses F16C instructions when available[1][9] |
OpenEXR's documentation notes that converting a float to half rounds the significand to 10 bits, losing its 13 least significant bits, and that converting half to float is lossless.[9] Imath rounds float-to-half conversion to nearest with ties to even.[1] Clang recommends _Float16 over __fp16 for portable code because the C standards committee defined it.[11]
Continue reading
- bfloat16bfloat16 (short for Brain Floating Point Format, sometimes written BF16) is a 16-bit floating-point number format that uses one sign bit, eight exponent bits, and seven mantissa bitsMentioned 8 times above
- Tensor CoreA Tensor Core is a specialized execution unit inside NVIDIA GPUs that computes a small matrix multiplication and accumulation, D = A x B + CMentioned 5 times above
- Mixed-Precision TrainingMixed-precision training is a technique for training deep learning models using lower-precision floating-point formats for most computations while maintaining a higher-precision copy of the model weights for…Mentioned twice above
- FP8FP8 is a family of 8-bit floating-point number formats used to accelerate deep learning training and inference, built around two encodings, E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2…Listed in See also
- FP4 (4-bit floating point)FP4 (4-bit floating point) is a numerical format that stores a real number in just 4 bits, the smallest floating-point type in mainstream use for deep learning.Listed in See also
- QuantizationQuantization is the representation of values from a large or continuous set by values from a smaller, usually finite set.Listed in See also
More to read
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Florian Kainz, Rod Bogart (primary original authors) and Contributors to the OpenEXR Project. "half.h". Imath, Academy Software Foundation (GitHub). github.com/...half.h
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12Nicholas J. Higham and Srikara Pranesh. "Simulating Low Precision Floating-Point Arithmetic". MIMS EPrint 2019.4, University of Manchester, July 2019 (published in SIAM Journal on Scientific Computing 41(5), C585-C602, 2019). eprints.maths.manchester.ac.uk/2723
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu. "Mixed Precision Training". ICLR 2018 (arXiv:1710.03740), 2017. arxiv.org/...1710.03740
- ^1 ^2 ^3BigScience Workshop. "BLOOM: A 176B-Parameter Open-Access Multilingual Language Model". arXiv:2211.05100, 2022. arxiv.org/...2211.05100
- ^1 ^2 ^3Shibo Wang and Pankaj Kanwar. "BFloat16: The secret to high performance on Cloud TPUs". Google Cloud Blog, 2019-08-23. cloud.google.com/...high-performance-on-cloud-tpus
- ^1 ^2vLLM. "Engine Arguments". vLLM documentation. docs.vllm.ai/...engine_args
- ^1 ^2Apple. "Typed Execution". Guide to Core ML Tools. apple.github.io/...typed-execution
- ^1 ^2Hugging Face. "Debugging". Transformers documentation. huggingface.co/...debugging
- ^1 ^2 ^3 ^4 ^5OpenEXR Project. "Technical Introduction to OpenEXR". OpenEXR documentation. openexr.com/...TechnicalIntroduction
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14Daniel Towner (author and editor). "Intel AVX-512 - FP16 Instruction Set for Intel Xeon Processor-Based Products". Intel Technology Guide, revision 004, March 2024. builders.intel.com/...hnology-guide-1710148129.pdf
- ^1 ^2 ^3 ^4 ^5Clang Team. "Clang Language Extensions". Clang documentation, LLVM Project. clang.llvm.org/...LanguageExtensions
- ^1 ^2OpenEXR Project. "About OpenEXR". OpenEXR documentation. openexr.com/...about
- ^1 ^2 ^3 ^4 ^5Mark Harris. "Mixed-Precision Programming with CUDA 8". NVIDIA Technical Blog, 2016-10-19. developer.nvidia.com/...ecision-programming-cuda-8
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8NVIDIA. "Train With Mixed Precision". NVIDIA Deep Learning Performance Documentation. docs.nvidia.com/...mixed-precision-training
- ^IEEE Standards Association. "IEEE 754-2019: IEEE Standard for Floating-Point Arithmetic". IEEE, 2019-07-22. standards.ieee.org/...6210
- ^Jeremy Appleyard and Scott Yokim. "Programming Tensor Cores in CUDA 9". NVIDIA Technical Blog, 2017-10-17. developer.nvidia.com/...amming-tensor-cores-cuda-9
- ^1 ^2 ^3Ronny Krashinsky, Olivier Giroux, Stephen Jones, Nick Stam, Sridhar Ramaswamy. "NVIDIA Ampere Architecture In-Depth". NVIDIA Technical Blog, 2020-05-14. developer.nvidia.com/...pere-architecture-in-depth
- ^1 ^2 ^3 ^4 ^5 ^6PyTorch. "Automatic Mixed Precision package - torch.amp". PyTorch 2.15 documentation. docs.pytorch.org/...amp
- ^1 ^2Mengdi Huang, Chetan Tekur, Michael Carilli. "Introducing native PyTorch automatic mixed precision for faster training on NVIDIA GPUs". PyTorch Blog, 2020-07-28. pytorch.org/...h-pytorch-automatic-mixed-precision
- ^Susan Zhang, Stephen Roller, Naman Goyal, et al. "OPT: Open Pre-trained Transformer Language Models". arXiv:2205.01068, 2022. arxiv.org/...2205.01068
- ^Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin. "Defeating the Training-Inference Mismatch via FP16". arXiv:2510.26788, 2025-10-30. arxiv.org/...2510.26788
- ^Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, et al. "A Study of BFLOAT16 for Deep Learning Training". arXiv:1905.12322, 2019. arxiv.org/...1905.12322
- ^cppreference.com. "Fixed width floating-point types (since C++23)". en.cppreference.com/...floating-point
- ^NumPy Developers. "Scalars". NumPy v2.5 Manual. numpy.org/...arrays.scalars
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
1 revision · v1 · 3,230 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Cite this page: AI Wiki. "FP16 (half precision)." aiwiki.ai, updated 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/fp16
