NVIDIA Hopper

NVIDIA Hopper is the codename for NVIDIA's ninth-generation datacenter GPU microarchitecture, announced March 22, 2022 by CEO Jensen Huang at the GTC keynote.^[1] Hopper succeeds the Ampere architecture used in the NVIDIA A100 and is the foundation for the NVIDIA H100, the NVIDIA H200, and the GH200 Grace Hopper Superchip.^[2] The architecture is named after Rear Admiral Grace Hopper, the U.S. Navy computer scientist who developed the first compiler and helped create COBOL.^[9]

Hopper was the workhorse of the generative AI boom from 2022 through 2024. Almost every frontier large language model trained during that period, including reported training runs for GPT-4, Claude 3, Gemini, and Llama 3.1 405B, ran on H100 clusters. The architecture introduced fourth-generation Tensor Cores with the FP8 numerical format, the Transformer Engine for dynamic precision selection, fourth-generation NVLink at 900 GB/s, the NVLink Switch System for 256-GPU domains, and the Tensor Memory Accelerator for asynchronous data movement.^[2] At launch NVIDIA quantified the generational gain as up to 9x faster training on a 395-billion-parameter Mixture-of-Experts model and up to 30x higher inference throughput on a Megatron 530B chatbot compared to A100.^[17] Hopper was eventually succeeded by NVIDIA Blackwell in 2024, but H100 and H200 inventory continues to dominate datacenter AI capacity well into 2026.

When was Hopper announced?

NVIDIA announced Hopper on March 22, 2022 during the GTC spring keynote. Jensen Huang revealed the H100 GPU as the first product based on the new GH100 die, manufactured by TSMC on a custom 4N process node (a 5nm-class derivative tuned specifically for NVIDIA).^[1] The H100 began shipping in volume in the third quarter of 2022 through SXM5 and PCIe form factors.^[3] Framing the launch, Huang said: "Data centers are becoming AI factories, processing and refining mountains of data to produce intelligence. NVIDIA H100 is the engine of the world's AI infrastructure that enterprises use to accelerate their AI-driven businesses."^[17]

The codename follows NVIDIA's tradition of naming microarchitectures after pioneering scientists. Earlier generations included Tesla, Fermi, Kepler, Maxwell, Pascal, Volta, Turing, and Ampere. Hopper sits between Ampere (A100, 2020) and Blackwell (B100/B200, 2024).^[9] The naming is paired with the Grace CPU, also developed by NVIDIA, which targets the Arm server market and combines with Hopper in the GH200 superchip module.^[5]

Hopper arrived at the moment that demand for large language model training was beginning to explode. By late 2022, after the launch of ChatGPT, every major AI lab and hyperscaler was attempting to acquire as many H100s as possible. The chip was effectively allocated rather than sold for most of 2023 and 2024, with reported lead times stretching past a year at the peak.

What products are in the Hopper family?

The Hopper family covers SXM and PCIe accelerators, an export-restricted China variant, and a CPU+GPU superchip module.

Scroll sideways for more →

The H100 SXM5 was the volume part for AI training and shipped on the HGX H100 baseboard, which carries eight GPUs interconnected through fourth-generation NVLink and NVSwitch.^[1] The H100 PCIe targeted enterprise servers that could not accept the higher 700 W power draw of SXM5. The H100 NVL, sometimes called the H100 NVL for Large Language Models, paired two PCIe cards over an NVLink bridge to deliver 188 GB of combined memory specifically for serving very large LLMs.^[3]

The H800 was a stripped-down variant created in response to U.S. export controls in October 2022, with reduced NVLink bandwidth so it could be sold legally into China.^[9] It was later joined by additional restricted SKUs as export rules evolved.

The H200, announced November 13, 2023 at SC23, used the same GH100 die as the H100 but added six 24 GB stacks of HBM3e memory in place of the five 16 GB HBM3 stacks on H100.^[6][10] That single change boosted capacity by 76 percent and bandwidth by 43 percent without any other architectural change.^[10] NVIDIA claimed roughly 1.9x the inference throughput on Llama 2 70B compared to H100 thanks to the larger working set and faster bandwidth.^[6]

The GH200 Grace Hopper Superchip integrates a Grace CPU (72 Arm Neoverse V2 cores, up to 480 GB LPDDR5X) with a Hopper GPU on a single module connected by NVLink-Chip-to-Chip (NVLink-C2C) at 900 GB/s coherent bandwidth, roughly seven times faster than PCIe Gen 5.^[5] The original GH200 used 96 GB of HBM3 on the GPU side; an updated GH200-141G announced in August 2023 added HBM3e to push the GPU memory to 144 GB.^[11]

Architecture and the GH100 die

Hopper is built around a single very large monolithic die called GH100, which set new records for transistor count and complexity in a datacenter GPU when it launched.^[2] The GH100 packs 80 billion transistors on an 814 mm squared die manufactured on the TSMC 4N process.^[1][2]

Scroll sideways for more →

Each H100 SM contains 128 FP32 cores and four fourth-generation Tensor Cores. The Tensor Cores are the workhorse of the chip for AI workloads and provide more than ten times the deep learning throughput of the standard CUDA cores.^[2] Hopper SMs include up to 228 KB of combined L1 cache and shared memory, about 1.33 times the per-SM capacity of A100, and they expose a new Distributed Shared Memory (DSMEM) capability that lets SMs in the same Thread Block Cluster read each other's L1 directly.^[2]

The GH100 die uses a 60 MB L2 cache, but ships products use 50 MB because one of the cache slices is paired with the disabled HBM stack.^[2] The L2 is split into two physically separated halves, which has interesting bandwidth and latency consequences that ChipsAndCheese and academic microbenchmark papers have analyzed in detail.^[14][15]

What makes Hopper different from Ampere?

Hopper is more than a process shrink of Ampere. It introduces several architectural features specifically designed for transformer-based workloads.

Scroll sideways for more →

The Transformer Engine deserves special attention. It is implemented partly in silicon (the FP8 Tensor Cores) and partly in the open-source Transformer Engine library, which integrates with PyTorch, JAX, and frameworks like Megatron-LM.^[8][16] The library tracks per-tensor scaling statistics during forward and backward passes and dynamically chooses between E4M3 (more mantissa bits, smaller range) for forward activations and E5M2 (more exponent bits, larger range) for gradients.^[8] The result is that large transformer models can train in FP8 for most operations while keeping a small fraction in FP16 or FP32 to preserve numerical stability.^[8] NVIDIA describes the Transformer Engine as a combination of "software and custom Hopper Tensor Core technology" that automatically handles re-casting and scaling between FP8 and 16-bit precision in each layer.^[8]

Thread Block Clusters and the Tensor Memory Accelerator together change how high-performance Hopper kernels are written. The TMA lets a single thread issue a multi-dimensional tile copy from HBM to shared memory while other warps continue computing, eliminating the traditional pattern of dedicating warps to address arithmetic and memory issue.^[2] Combined with asynchronous transaction barriers, this enables the producer-consumer pipeline patterns that show up in libraries like CUTLASS 3.x and FlashAttention-3.

How fast is the H100?

The published H100 SXM5 performance numbers from the NVIDIA datasheet are reproduced below. These are peak theoretical throughputs and the FP8/FP16/BF16/TF32/INT8 figures include the structured 2:4 sparsity feature.^[3]

Scroll sideways for more →

Real-world FP8 performance during training typically lands in the 700 to 1,200 TFLOPS range per GPU depending on the workload, network topology, and how aggressively the Transformer Engine can stay in FP8. Meta reported around 380 teraFLOP/s sustained per GPU during the Llama 3.1 405B training run on a 16,384 H100 cluster, which corresponds to roughly 38 percent of dense FP8 peak.^[13]

How does Hopper compare with Ampere (A100)?

Hopper is roughly a generational leap over the A100, the previous datacenter flagship. The improvements come from three independent factors stacking together: more SMs, faster Tensor Cores per SM, and a new lower-precision format (FP8) that doubles arithmetic density.

Scroll sideways for more →

NVIDIA's headline claims of 9x faster training and up to 30x faster LLM inference for H100 vs A100 depend on FP8 being usable for the workload and on the model being large enough to benefit from the larger memory capacity.^[1][17] For HPC FP64 workloads the improvement is more modest, in the 3x to 4x range. For dense LLM inference of very large models, the combination of HBM3, larger L2, the Transformer Engine, and faster NVLink can produce more than 10x throughput improvements at equivalent latency.

How does Hopper compare with Blackwell?

Blackwell, introduced in March 2024, is Hopper's successor. Blackwell uses a dual-die package (two reticle-sized chips connected by a 10 TB/s die-to-die link) and adds support for FP4 and a refined FP8 implementation. It targets a roughly 2x to 4x performance improvement over Hopper on most LLM workloads, depending on whether FP4 is used.

Scroll sideways for more →

Despite Blackwell's launch, Hopper continues to ship in volume through 2025 and 2026 because demand for AI compute massively exceeds supply and because TSMC CoWoS advanced packaging capacity is the binding constraint on Blackwell production. H100 and H200 remain the most commonly available datacenter accelerator in cloud catalogs at the time of writing.

Software stack

Hopper requires CUDA 11.8 or later for basic support and CUDA 12.x for full feature support including FP8 Tensor Cores, Thread Block Clusters, the TMA, and the DPX instructions.^[2] The full software ecosystem includes:

Scroll sideways for more →

The Transformer Engine library is open source and lives at github.com/NVIDIA/TransformerEngine.^[16] It integrates with PyTorch through te.LayerNormLinear, te.MultiheadAttention, and similar drop-in modules, and with JAX through Praxis. FlashAttention-3, released in 2024, was rewritten specifically to take advantage of Hopper TMA and asynchronous warp specialization, and pushed attention throughput on H100 to roughly 75 percent of FP16 Tensor Core peak.

Which models were trained on Hopper?

Hopper became the dominant training accelerator for frontier AI models from 2022 through 2024. Most training clusters are reported in approximate H100 counts, since the exact numbers are usually company-confidential.

Scroll sideways for more →

The Llama 3.1 405B training run is the most thoroughly documented, since Meta published its technical report.^[13] The 16,384-GPU cluster sustained around 380 teraFLOP/s per H100 in BF16 and experienced roughly one component failure every three hours, with HBM3 and GPU faults accounting for about half of all failures.^[12]

Datacenter deployment

Hopper ships in several reference platforms designed for different scales of deployment.

Scroll sideways for more →

Major cloud providers have all built large H100 fleets. AWS offers H100 through P5 instances (eight H100 SXM per instance). Microsoft Azure offers ND H100 v5 series. Google Cloud offers A3 instances. Oracle Cloud Infrastructure built one of the largest H100 superclusters for OpenAI's training. CoreWeave and Lambda Labs operate H100 clouds focused on AI workloads. By 2024 the secondary GPU cloud market had grown to hundreds of operators reselling H100 capacity.

How much does an H100 cost?

H100 supply was the binding constraint on AI development through 2023 and most of 2024. Industry reporting from Raymond James and others in late 2023 and early 2024 placed the per-GPU price for H100 SXM at roughly $25,000 to $30,000 with secondary market prices reportedly above $40,000. Complete DGX H100 systems were widely cited as costing $400,000 to $500,000.

Lead times for direct H100 orders from NVIDIA reached six months to over a year at the peak. Cloud provider allocation became a major business consideration; reports suggested that H100 access drove material revenue at AWS, Azure, GCP, and Oracle Cloud, and was a competitive differentiator for AI-focused clouds like CoreWeave. Several large AI labs disclosed multi-billion dollar H100 commitments. The supply constraint shifted upstream to TSMC's CoWoS advanced packaging capacity and to SK Hynix and Micron HBM3/HBM3e production, both of which became their own well-publicized bottlenecks.

Limitations

Hopper has several well-understood limitations that drove the design of Blackwell. The 700 W TDP per SXM5 module pushed datacenter cooling to its limits and effectively required liquid cooling at rack densities above 40 kW per rack.^[1] Even at lower densities, the power and thermal envelope of an HGX H100 server (roughly 10 kW for the GPU portion alone) requires careful facility design.

The chip is monolithic and at 814 mm squared sits near the reticle limit of TSMC's lithography tools, which constrains how much further single-die scaling can go.^[2] Blackwell's response was to move to a dual-die package. Memory capacity per GPU was a frequent bottleneck for serving very large models; H100's 80 GB forced the use of tensor parallelism across multiple GPUs even for inference of 70B-class models, which is why H200's jump to 141 GB and the GH200's 480 GB of CPU memory were so well received.^[6]

HBM supply was the most-cited single constraint on H100 production. The chip uses five active stacks of HBM3 per GPU, and the total industry HBM3 output through 2023 and 2024 was insufficient to meet H100 demand, let alone leave headroom for the H200 and Blackwell ramp.^[2]

Finally, FP8 software tooling was immature in the first year after launch. Production training in FP8 only became routine in late 2023 and 2024 as the Transformer Engine library matured, FlashAttention-3 landed, and major training frameworks (Megatron-LM, NeMo, MosaicML/Composer) shipped robust FP8 paths.

References

1. NVIDIA. "NVIDIA H100 Tensor Core GPU Architecture" whitepaper, March 2022. https://resources.nvidia.com/en-us-hopper-architecture/nvidia-h100-tensor-c ↩
2. NVIDIA Developer Blog. "NVIDIA Hopper Architecture In-Depth." https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ ↩
3. NVIDIA. "H100 Tensor Core GPU" product page and datasheet. https://www.nvidia.com/en-us/data-center/h100/ ↩
4. NVIDIA. "H200 Tensor Core GPU" product page and datasheet. https://www.nvidia.com/en-us/data-center/h200/
5. NVIDIA. "GH200 Grace Hopper Superchip" product page. https://www.nvidia.com/en-us/data-center/grace-hopper-superchip/ ↩
6. NVIDIA Newsroom. "NVIDIA Supercharges Hopper, the World's Leading AI Computing Platform." November 13, 2023. https://nvidianews.nvidia.com/news/nvidia-supercharges-hopper-the-worlds-leading-ai-computing-platform ↩
7. NVIDIA Investor Relations. "NVIDIA Grace Hopper Superchips Designed for Accelerated Generative AI Enter Full Production." May 28, 2023.
8. NVIDIA Blog. "H100 Transformer Engine Supercharges AI Training." https://blogs.nvidia.com/blog/h100-transformer-engine/ ↩
9. Wikipedia. "Hopper (microarchitecture)." https://en.wikipedia.org/wiki/Hopper_(microarchitecture) ↩
10. Tom's Hardware. "Nvidia Announces H200 GPU: 141GB of HBM3e and 4.8 TB/s Bandwidth." November 13, 2023. https://www.tomshardware.com/news/nvidia-h200-gpu-announced ↩
11. AnandTech. "NVIDIA Unveils Updated GH200 Grace Hopper Superchip with HBM3e Memory." August 2023. https://www.anandtech.com/show/20001/nvidia-unveils-gh200-grace-hopper-gpu-with-hbm3e-memory ↩
12. Tom's Hardware. "Faulty Nvidia H100 GPUs and HBM3 memory caused half of failures during Llama 3 training." ↩
13. Meta AI. "Introducing Llama 3.1: Our Most Capable Models to Date." https://ai.meta.com/blog/meta-llama-3-1/ ↩
14. ChipsAndCheese. "Nvidia's H100: Funny L2, and Tons of Bandwidth." https://chipsandcheese.com/p/nvidias-h100-funny-l2-and-tons-of-bandwidth ↩
15. Luo et al., "Benchmarking and Dissecting the Nvidia Hopper GPU Architecture." arXiv:2402.13499. https://arxiv.org/html/2402.13499v1 ↩
16. NVIDIA. Transformer Engine library on GitHub. https://github.com/NVIDIA/TransformerEngine ↩
17. NVIDIA Newsroom. "NVIDIA Announces Hopper Architecture, the Next Generation of Accelerated Computing." March 22, 2022. https://nvidianews.nvidia.com/news/nvidia-announces-hopper-architecture-the-next-generation-of-accelerated-computing ↩