FP4 (4-bit floating point)
FP4 (4-bit floating point) is a numerical format that stores a real number in just 4 bits, the smallest floating-point type in mainstream use for deep learning.
Explore Training & Optimization through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of Training & Optimization.
Showing 1-5 of 5 articles
FP4 (4-bit floating point) is a numerical format that stores a real number in just 4 bits, the smallest floating-point type in mainstream use for deep learning.
FP8 is a family of 8-bit floating-point number formats used to accelerate deep learning training and inference, built around two encodings, E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2…
MXFP4 (Microscaling FP4) is a 4-bit block-scaled number format for quantization in machine learning, defined in the Open Compute Project (OCP) Microscaling Formats (MX) Specification v1.0 of September 2023.
NCCL, the NVIDIA Collective Communications Library, is a library of topology-aware communication primitives for NVIDIA GPU systems.
A Tensor Core is a specialized execution unit inside NVIDIA GPUs that computes a small matrix multiplication and accumulation, D = A x B + C