Citation and evidence

FP16 (half precision)

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

AI HardwareComputer ScienceDeep Learning

Cite this article

FP16 (also called half precision, binary16 or float16) is a 16-bit floating-point number format with one sign bit, five exponent bits and ten stored significand bits, specified as the binary16 format of the IEEE 754 standard for floating-point arithmetic.[1][2] It takes half the memory of 32-bit single precision (FP32), and in deep learning it was the 16-bit format of the mixed-precision training method that NVIDIA and Baidu researchers published in 2017, which showed that neural networks trained with FP16 storage and arithmetic, an FP32 master copy of the weights and loss scaling could match FP32 accuracy.[3] Its narrow range (the largest finite value is 65,504) is its main weakness, and later large models such as BLOOM were trained in bfloat16, a 16-bit format that keeps FP32's eight-bit exponent.[4][5] FP16 is still used for inference and in graphics and signal processing.[6][7][9][10]

Format

Bit layout

An FP16 value is a 16-bit word split into three fields.[1]

BitsFieldWidthRole
15Sign (s)1 bit0 for positive, 1 for negative
14-10Exponent (e)5 bitsBiased exponent, bias 15
9-0Significand (m)10 bitsFraction bits after the binary point

Expanded article table

The stored exponent field $e$ selects how the bits are read:[1]

  • If $1 \le e \le 30$, the number is normal: $x = (-1)^s \times 2^{e-15} \times 1.m$, where $1.m$ is the binary fraction with an implicit leading 1.
  • If $e = 0$ and $m \neq 0$, the number is subnormal (denormalized): $x = (-1)^s \times 2^{-14} \times 0.m$.
  • If $e = 0$ and $m = 0$, the value is zero (positive or negative, depending on the sign bit).
  • If $e = 31$, the value is infinity when $m = 0$ and NaN (not a number) otherwise.

With the implicit bit, a normal FP16 number carries 11 bits of significand precision, and normal values have exponents from -14 to 15.[3][11] The Imath library, whose half type was formerly distributed with OpenEXR as part of IlmBase, documents this layout and describes the type as compatible with the IEEE 754-2008 binary16 type.[1][12]

Notable values

ValueBit pattern (hex)Approximate decimal
Smallest positive subnormal ($2^{-24}$)0x00015.96e-8[10]
Largest subnormal0x03FF6.0976e-5[10]
Smallest positive normal ($2^{-14}$)0x04006.10e-5[10]
One0x3C001[10]
Largest finite value0x7BFF65,504[1]
Positive infinity0x7C00Inf[10]
Negative infinity0xFC00-Inf[10]

Expanded article table

The Imath header encodes the most negative finite value, -65,504, as 0xFBFF; clearing the sign bit gives 0x7BFF.[1]

The machine epsilon, the gap between 1 and the next larger FP16 number, is $2^{-10} \approx 0.00097656$, the value Imath defines as HALF_EPSILON, and Imath counts three decimal digits that survive a round trip through the format unchanged.[1] Higham and Pranesh give its unit roundoff as $4.88 \times 10^{-4}$.[2] Every integer up to 2048 in magnitude can be represented exactly; 2049 is the first integer that cannot be, and above 2048 the representable integers thin out quickly.[10] Between $2^{-14}$ and $2^{15}$ FP16 can express about 30,720 distinct values.[13]

Range and its consequences

NVIDIA's mixed-precision guide describes FP16's dynamic range, counting subnormals, as 40 powers of two, against 264 for FP32.[14] Intel's guide points out three practical effects of that small range: subnormal numbers are "not very small", so values quickly fall below what normal FP16 numbers can hold; infinity begins only slightly below the largest unsigned 16-bit integer, so converting 16-bit integers can overflow; and rounding integers above 2048 to FP16 introduces large absolute errors.[10] In deep learning, the narrow range shows up as overflow in large activations and as small gradients rounding to zero.[3][8][14]

History

Graphics origins

Sixteen-bit floating point also has a history in computer graphics. OpenEXR, an image file format developed at Industrial Light & Magic (ILM) and first released as open source in 2003, stores HALF channels with the 1-5-10 layout, supporting normal and subnormal numbers, infinities and NaNs.[9][12] Its documentation states that its half class is identical to NVIDIA's 16-bit floating-point format (fp16/half), so 16-bit data can move between OpenEXR files and NVIDIA 16-bit frame buffers without losing bits.[9] The Imath header names Florian Kainz and Rod Bogart of ILM as the primary original authors of the half type.[1]

Standardization

For about two decades after the 1985 IEEE 754 standard, virtually all floating-point hardware implemented its single (fp32) or double (fp64) precision. The 2008 revision of the standard defined a 16-bit half precision format, but as a storage format: it did not define arithmetic operations on it.[2] Hardware vendors nonetheless implemented half-precision arithmetic by extending the rules used for single and double precision.[2] The current edition, IEEE 754-2019, was approved on 2019-06-13 and superseded the 2008 edition; the IEEE also lists ISO/IEC/IEEE 60559:2020, an international floating-point standard with the same scope, among the 754 working group's active standards.[15] Intel describes its AVX-512 FP16 instructions as compliant with IEEE 754-2019, the edition it calls the current standard.[10]

Hardware support

YearHardwareFP16 capability
2016NVIDIA Tesla P100 (Pascal)FP16 arithmetic at twice the FP32 rate, 21.2 TFLOPS on the SXM2 module[13]
2017NVIDIA V100 (Volta)640 Tensor Cores multiplying FP16 matrices with FP16 or FP32 accumulation, up to 125 Tensor TFLOPS[16]
2017AMD Radeon Instinct MI25Half-precision arithmetic[2]
2020NVIDIA A100 (Ampere)312 TFLOPS peak FP16 Tensor Core throughput, with BF16 at the same rate[17]

Expanded article table

The P100 reaches its FP16 peak only with paired half2 instructions, which operate on two FP16 values packed into one 32-bit register; scalar half instructions run at about half of peak.[13] Other Pascal chips (GP102, GP104 and GP106) ran FP16 at a lower rate, similar to their double-precision rate.[13] Higham and Pranesh also list Arm's NEON architecture among the platforms with half-precision arithmetic.[2] Clang reports native _Float16 arithmetic on AArch64 from ARMv8.2-A, on AMD and NVIDIA GPU targets, and on x86 when AVX512-FP16 is available.[11]

On x86 processors, Intel's 3rd generation Core processors and their successors could only convert between half precision and FP32, so numeric work had to be done in a higher precision.[10] Intel then added an AVX-512 FP16 instruction set to its 4th generation Xeon Scalable processors, with general-purpose FP16 arithmetic, fused multiply-add, comparisons, conversions and native complex-number support.[10] Intel's guide to these instructions (first revision October 2021) says FP16 handles twice as many operations per clock cycle as FP32 and four times as many as FP64, and it lists signal processing, media and video processing, artificial intelligence and machine learning as typical uses.[10]

Higham and Pranesh cite one example of what the format offers outside machine learning: the Summit supercomputer at Oak Ridge National Laboratory had a peak of 148.6 petaflops on the double-precision LINPACK benchmark, but reached 2.36 exaflops on a genetics application that used half precision.[2]

Use in deep learning

Mixed precision training

In "Mixed Precision Training", published at ICLR 2018 by Paulius Micikevicius and colleagues from NVIDIA and Baidu Research, weights, activations and gradients are stored in FP16, and three techniques keep accuracy at the FP32 level:[3]

  1. FP32 master weights. An FP32 copy of the weights receives the updates in the optimizer step, and an FP16 copy is used in the forward and backward passes. The paper gives two reasons. Updates smaller than $2^{-24}$ become zero in FP16; in a Mandarin speech model, about 5% of weight-gradient values had exponents below -24. And an update can vanish when it is added to a weight at least 2048 times larger, because aligning the binary points shifts it out of the 10-bit significand. Updating FP16 weights directly in the speech model caused an 80% relative accuracy loss.[3]
  2. Loss scaling. The loss is multiplied by a constant before backpropagation, so every gradient is scaled up by the chain rule and fewer of them fall below the FP16 range.[3]
  3. FP32 accumulation. FP16 dot products accumulate partial products in FP32 and convert back to FP16 when writing to memory, and large reductions, such as batch-normalization statistics and softmax, are computed in FP32.[3]

The extra FP32 weight copy raises weight memory by 50%, but because activations dominate training memory and are stored in FP16, overall memory use is roughly halved.[3] The authors reported matching FP32 results on convolutional and recurrent networks and GANs, including models with more than 100 million parameters.[3] See mixed-precision training for the method in general, including its later BF16 and FP8 forms.

Loss scaling

The scaling problem is concrete. In the Multibox SSD detector, converting activation gradients to FP16 turned 31% of the values into zeros, leaving 5.3% nonzero, and training diverged.[14] Multiplying the gradients by 8 (shifting exponents by three) matched FP32 accuracy, and the paper concluded that activation-gradient values below $2^{-27}$ did not matter for that model while values between $2^{-27}$ and $2^{-24}$ had to be kept.[3][14]

Gradients are divided by the scale factor before the weight update, preferably right after the backward pass and before gradient clipping, so hyperparameters such as the clipping threshold and weight decay need no change.[3] The NVIDIA and Baidu paper and NVIDIA's guide describe two ways of choosing the factor:

ApproachHow it worksReported settings
Static (constant)A fixed factor, chosen by trial or so that the factor times the largest gradient stays below 65,504Factors from 8 to 32K across tested networks; many needed none[3]
DynamicStart large; skip the update and shrink the factor when an overflow (Inf or NaN) appears; grow it after a run of overflow-free stepsNVIDIA's runs used N = 2000 steps, growth factor 2, reduction factor 0.5[14]

Expanded article table

PyTorch's GradScaler implements dynamic scaling with defaults of an initial scale of 65,536, a growth factor of 2, a backoff factor of 0.5 and a growth interval of 2000 steps.[18] The PyTorch documentation explains the need for it directly: small gradient values may not be representable in float16, flush to zero, and the corresponding parameter updates are lost.[18]

Frameworks and automatic mixed precision

NVIDIA released Apex, a PyTorch extension with an automatic mixed precision (AMP) feature, in 2018; for PyTorch 1.6 (2020), developers at NVIDIA and Facebook moved the functionality into PyTorch itself as torch.cuda.amp, and Apex AMP went into maintenance mode.[19] In benchmarks published with that release, AMP with FP16 trained models 1.5 to 5.5 times faster than FP32 on V100 GPUs while reaching the same final accuracy.[19]

In current PyTorch, torch.autocast chooses a precision per operation. On CUDA, operations such as matmul, linear, conv2d and bmm run in float16, while softmax, layer_norm, exp, cross_entropy and sum are promoted to float32.[18] NVIDIA's guide sorts operations the same way: convolutions and fully connected layers run in FP16, large reductions, cross-entropy, L1 loss and exponentials stay in FP32, and element-wise operations follow their inputs.[14]

To use Tensor Cores with FP16 inputs, NVIDIA's guide asks that matrix dimensions (M, N and K) and convolution channel counts be multiples of 8, and recommends padding vocabulary size and sequence length to multiples of 8.[14] The guide quotes overall speedups of up to 3x on the most arithmetically intense models.[14]

Training large language models

Some early large language models were trained with FP16 weights. Meta AI's OPT-175B kept Adam optimizer state in FP32 while the model weights stayed in FP16, and used dynamic loss scaling to avoid underflow; when the loss diverged, the team restarted from points where the dynamic loss scaler was still at or above 1.0, and resetting the scaler helped recover some but not all divergences.[20]

The BLOOM team reported irreversible divergences in earlier 104-billion-parameter experiments on V100 GPUs, which it attributed to float16's limited dynamic range; the final 176B model was trained in bfloat16 mixed precision on A100 GPUs, which the paper says solved the instability problem.[4] The BLOOM paper also notes that float16 has since been held responsible for many of the instabilities observed in LLM training.[4] The PyTorch documentation gives a related warning: most models pretrained in bf16 cannot operate within FP16's maximum of 65,504, so their gradients overflow rather than underflow, and NaNs in the loss or gradients under FP16 mixed precision are a sign that the model may not be compatible.[18] Hugging Face's debugging guide walks through an FP16 overflow in google/mt5-small, where a feed-forward output reached about 6.27e4, near the FP16 maximum, and dropout's rescaling then pushed values to infinity.[8]

Inference

Some inference tools default to FP16. The vLLM inference engine, with its default auto data type, runs FP32 and FP16 checkpoints in FP16 and BF16 checkpoints in BF16.[6] Apple's Core ML converter types an ML program in float16 precision by default. For models in the older neural network format, Apple's documentation says the GPU and Neural Engine run in float16 while the CPU runs in float32; in ML programs the GPU supports both precisions and some operations, such as convolution, can also run in float16 on the CPU.[7] A 2025 preprint by Penghui Qi and colleagues argued that FP16 can also help reinforcement-learning fine-tuning of LLMs: in their account, BF16's large rounding errors break the consistency between the training and inference computations, and switching to FP16 removed that mismatch and gave more stable optimization.[21]

Comparison with bfloat16

bfloat16 (BF16) spends its 16 bits differently: 8 exponent bits and 7 stored significand bits, which gives it FP32's range at much lower precision.[2][5]

PropertyFP16BF16FP32
Exponent bits588
Significand bits (with implicit bit)11824
Largest finite value6.55e43.39e383.40e38
Smallest normal value6.10e-51.18e-381.18e-38
Smallest subnormal value5.96e-89.18e-411.40e-45
Unit roundoff4.88e-43.91e-35.96e-8

Expanded article table

Values from Higham and Pranesh's comparison table, rounded to three significant figures.[2] The BF16 subnormal figure applies only where subnormals are supported; Intel's bfloat16 specification flushes them to zero.[2]

The table shows the trade. FP16's unit roundoff is eight times smaller than BF16's, so each FP16 value is more precise, but FP16 covers a far smaller range of magnitudes.[2] Google's Shibo Wang and Pankaj Kanwar wrote in 2019 that FP16 "typically requires special handling via techniques such as loss scaling", while bfloat16 is close to a drop-in replacement for FP32; they also note that a hardware multiplier's area grows with the square of the significand width, which makes a bfloat16 multiplier about half the size of a typical FP16 one.[5] A 2019 study by Dhiraj Kalamkar and colleagues trained image, speech, language, generative and recommendation models in emulated BF16 and reported results comparable to FP32 without changing hyperparameters, in contrast to the tuning FP16 typically needs.[22]

Intel's FP16 guide describes the difference in purpose: bfloat16 is the upper 16 bits of FP32 and suits machine learning, while IEEE FP16 balances precision and range in a way it considers better for general-purpose algorithms.[10] On NVIDIA's A100, BF16 and FP16 Tensor Core operations run at the same rate (312 TFLOPS dense), although outside the Tensor Cores NVIDIA lists peak FP16 at 78 TFLOPS against 39 TFLOPS for BF16.[17] Moving a model between the formats is asymmetric: BF16-trained models can overflow when run in FP16, while every finite FP16 magnitude lies inside BF16's range, although converting to BF16 loses precision.[2][18]

NVIDIA's TensorFloat-32 (TF32) mode on Ampere GPUs combines the two designs, pairing FP32's 8-bit exponent with FP16's 10-bit mantissa.[17]

Programming language and library support

EnvironmentFP16 typeNotes
C (ISO/IEC TS 18661-3:2015)_Float16Interchange floating type; Clang supports it on many targets[11]
Clang (Arm extension)__fp16Originally an ARM extension; never used directly for arithmetic, operands promoted to float; Clang uses IEEE binary16 rather than Arm's alternative half-precision format[11]
C++23std::float16_t in <stdfloat>Optional; available when __STDCPP_FLOAT16_T__ is defined, literal suffix f16[23]
CUDAhalf, half2 in cuda_fp16.hIntrinsics such as __hfma and __hadd2[13]
NumPynumpy.float16 (numpy.half)Character code 'e'[24]
PyTorchtorch.float16 (half)Used by torch.autocast and GradScaler[18]
Imath / OpenEXRhalfC++ class that mixes with float and double; conversion uses F16C instructions when available[1][9]

Expanded article table

OpenEXR's documentation notes that converting a float to half rounds the significand to 10 bits, losing its 13 least significant bits, and that converting half to float is lossless.[9] Imath rounds float-to-half conversion to nearest with ties to even.[1] Clang recommends _Float16 over __fp16 for portable code because the C standards committee defined it.[11]

Continue reading

More to read

Topics:AI HardwareComputer ScienceDeep Learning

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Florian Kainz, Rod Bogart (primary original authors) and Contributors to the OpenEXR Project. "half.h". Imath, Academy Software Foundation (GitHub). github.com/...half.h
  2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12Nicholas J. Higham and Srikara Pranesh. "Simulating Low Precision Floating-Point Arithmetic". MIMS EPrint 2019.4, University of Manchester, July 2019 (published in SIAM Journal on Scientific Computing 41(5), C585-C602, 2019). eprints.maths.manchester.ac.uk/2723
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu. "Mixed Precision Training". ICLR 2018 (arXiv:1710.03740), 2017. arxiv.org/...1710.03740
  4. ^1 ^2 ^3BigScience Workshop. "BLOOM: A 176B-Parameter Open-Access Multilingual Language Model". arXiv:2211.05100, 2022. arxiv.org/...2211.05100
  5. ^1 ^2 ^3Shibo Wang and Pankaj Kanwar. "BFloat16: The secret to high performance on Cloud TPUs". Google Cloud Blog, 2019-08-23. cloud.google.com/...high-performance-on-cloud-tpus
  6. ^1 ^2vLLM. "Engine Arguments". vLLM documentation. docs.vllm.ai/...engine_args
  7. ^1 ^2Apple. "Typed Execution". Guide to Core ML Tools. apple.github.io/...typed-execution
  8. ^1 ^2Hugging Face. "Debugging". Transformers documentation. huggingface.co/...debugging
  9. ^1 ^2 ^3 ^4 ^5OpenEXR Project. "Technical Introduction to OpenEXR". OpenEXR documentation. openexr.com/...TechnicalIntroduction
  10. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14Daniel Towner (author and editor). "Intel AVX-512 - FP16 Instruction Set for Intel Xeon Processor-Based Products". Intel Technology Guide, revision 004, March 2024. builders.intel.com/...hnology-guide-1710148129.pdf
  11. ^1 ^2 ^3 ^4 ^5Clang Team. "Clang Language Extensions". Clang documentation, LLVM Project. clang.llvm.org/...LanguageExtensions
  12. ^1 ^2OpenEXR Project. "About OpenEXR". OpenEXR documentation. openexr.com/...about
  13. ^1 ^2 ^3 ^4 ^5Mark Harris. "Mixed-Precision Programming with CUDA 8". NVIDIA Technical Blog, 2016-10-19. developer.nvidia.com/...ecision-programming-cuda-8
  14. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8NVIDIA. "Train With Mixed Precision". NVIDIA Deep Learning Performance Documentation. docs.nvidia.com/...mixed-precision-training
  15. ^IEEE Standards Association. "IEEE 754-2019: IEEE Standard for Floating-Point Arithmetic". IEEE, 2019-07-22. standards.ieee.org/...6210
  16. ^Jeremy Appleyard and Scott Yokim. "Programming Tensor Cores in CUDA 9". NVIDIA Technical Blog, 2017-10-17. developer.nvidia.com/...amming-tensor-cores-cuda-9
  17. ^1 ^2 ^3Ronny Krashinsky, Olivier Giroux, Stephen Jones, Nick Stam, Sridhar Ramaswamy. "NVIDIA Ampere Architecture In-Depth". NVIDIA Technical Blog, 2020-05-14. developer.nvidia.com/...pere-architecture-in-depth
  18. ^1 ^2 ^3 ^4 ^5 ^6PyTorch. "Automatic Mixed Precision package - torch.amp". PyTorch 2.15 documentation. docs.pytorch.org/...amp
  19. ^1 ^2Mengdi Huang, Chetan Tekur, Michael Carilli. "Introducing native PyTorch automatic mixed precision for faster training on NVIDIA GPUs". PyTorch Blog, 2020-07-28. pytorch.org/...h-pytorch-automatic-mixed-precision
  20. ^Susan Zhang, Stephen Roller, Naman Goyal, et al. "OPT: Open Pre-trained Transformer Language Models". arXiv:2205.01068, 2022. arxiv.org/...2205.01068
  21. ^Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin. "Defeating the Training-Inference Mismatch via FP16". arXiv:2510.26788, 2025-10-30. arxiv.org/...2510.26788
  22. ^Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, et al. "A Study of BFLOAT16 for Deep Learning Training". arXiv:1905.12322, 2019. arxiv.org/...1905.12322
  23. ^cppreference.com. "Fixed width floating-point types (since C++23)". en.cppreference.com/...floating-point
  24. ^NumPy Developers. "Scalars". NumPy v2.5 Manual. numpy.org/...arrays.scalars

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

1 revision · v1 · 3,230 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Cite this page: AI Wiki. "FP16 (half precision)." aiwiki.ai, updated 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/fp16

Edit article