# TPU Pod

> Source: https://aiwiki.ai/wiki/tpu_pod
> Summary: A TPU Pod is a single Google supercomputer built from many Tensor Processing Unit (TPU) chips wired directly to each other by a high-speed Inter-Chip Interconnect (ICI) fabric arranged as a 2D or 3D torus, so that software can treat the whole cluster as a single unit for machine learning training…
> Updated: 2026-10-15
> Fact-checked: 2026-10-15
> Categories: AI Compute, AI Hardware, AI Infrastructure, Google, Machine Learning
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "TPU Pod." aiwiki.ai, 15 Oct 2026. https://aiwiki.ai/wiki/tpu_pod
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

*See also: [Tensor Processing Unit (TPU)](https://aiwiki.ai/wiki/tpu), [Cloud TPU](https://aiwiki.ai/wiki/cloud_tpu), [Data parallelism](https://aiwiki.ai/wiki/data_parallelism), [Model parallelism](https://aiwiki.ai/wiki/model_parallelism), [Distributed training](https://aiwiki.ai/wiki/distributed_training)*

## Overview

A **TPU Pod** is a single [Google](https://aiwiki.ai/wiki/google) supercomputer built from many Tensor Processing Unit (TPU) chips wired directly to each other by a high-speed Inter-Chip Interconnect (ICI) fabric arranged as a 2D or 3D torus, so that software can treat the whole cluster as a single unit for [machine learning](https://aiwiki.ai/wiki/machine_learning) training and [inference](https://aiwiki.ai/wiki/inference).[7][17] Pod size has grown across Google's performance-class TPUs: a [TPU v4](https://aiwiki.ai/wiki/google_tpu_v4) Pod holds 4,096 chips, a [TPU v5p](https://aiwiki.ai/wiki/google_tpu_v5p) Pod holds 8,960 chips, and the seventh-generation [Ironwood (TPU v7)](https://aiwiki.ai/wiki/tpu_ironwood) superpod holds 9,216 chips delivering 42.5 exaFLOPS of FP8 compute, the largest TPU Pod available as of October 2026; Google announced 9,600-chip superpods for its eighth-generation [TPU 8t](https://aiwiki.ai/wiki/google_tpu_8t) in April 2026.[2][9][12][23] TPU Pods are the machines on which Google trains its largest models, including [PaLM](https://aiwiki.ai/wiki/palm) and the [Gemini](https://aiwiki.ai/wiki/gemini) family.[4][35]

Google describes the top Ironwood configuration in concrete terms: "When scaled to 9,216 chips per pod for a total of 42.5 Exaflops, Ironwood supports more than 24x the compute power of the world's largest supercomputer," which Google identified as El Capitan.[12] That comparison sets Ironwood's FP8 peak against El Capitan's 64-bit LINPACK result.[18] Rather than treating each chip as a separate device, a TPU Pod lets the [XLA](https://aiwiki.ai/wiki/xla) compiler view the entire cluster as one unified computational resource, enabling efficient distributed training of [large language models](https://aiwiki.ai/wiki/large_language_model) and other compute-intensive AI systems.

Google introduced the TPU Pod concept with TPU v2 in 2017, when the company first connected multiple TPU chips via custom ICI links arranged in a 2D torus topology.[14] Since then, each performance-class TPU generation has expanded Pod scale and interconnect sophistication. Early Pods contained 256 chips; Ironwood (TPU v7), the newest generally available generation, scales to 9,216 chips per Pod and delivers 42.5 exaFLOPS of aggregate FP8 compute.[12] TPU Pods have been used to train many of the most influential [AI models](https://aiwiki.ai/wiki/model) of the past decade, including PaLM, Gemini, and [BERT](https://aiwiki.ai/wiki/bert).[4][31][35]

A TPU Pod differs from a conventional [GPU](https://aiwiki.ai/wiki/gpu) cluster in a fundamental way: the chips communicate through a dedicated, low-latency torus network rather than through general-purpose [data center](https://aiwiki.ai/wiki/data_center) switches. This design enables collective operations (such as [all-reduce](https://aiwiki.ai/wiki/all_reduce)) to run with minimal overhead, which is particularly important when thousands of chips must synchronize gradient updates during training.

## ELI5 (Explain like I'm 5)

Imagine you have a giant jigsaw puzzle with millions of pieces. If you try to solve it by yourself, it could take weeks. But what if you got a whole classroom of friends to help? Each friend works on a section of the puzzle, and whenever they need a piece from someone else's section, they can pass it over quickly because they are all sitting at the same big table.

A TPU Pod works the same way. Each [TPU chip](https://aiwiki.ai/wiki/tpu_chip) is like one of those friends, and the special wires connecting them (called the "Inter-Chip Interconnect") are like the table that lets them pass puzzle pieces to each other really fast. Because the chips are all connected directly to their neighbors, they can share information almost instantly, so the whole group finishes the puzzle much faster than any single chip could on its own.

## How did the TPU Pod evolve across generations?

### TPU v1: single-chip inference (2015)

The first TPU was an inference-only chip with no inter-chip network. Each TPU v1 sat on its own PCIe card inside a server, operating independently.[1] There was no concept of a "Pod" at this stage; the v1 was designed to accelerate [neural network](https://aiwiki.ai/wiki/neural_network) inference for production services such as [Google Search](https://aiwiki.ai/wiki/google_search), Google Translate, and the [AlphaGo](https://aiwiki.ai/wiki/alphago) system.[1]

### TPU v2: the first Pods (2017)

Announced at Google I/O in May 2017, TPU v2 was the first generation to support both training and inference.[14] It was also the first to introduce the Inter-Chip Interconnect (ICI), a custom high-speed bidirectional link connecting each chip directly to four neighbors in a **2D torus** topology. Groups of four chips were packaged into modules delivering 180 TFLOPS. Sixty-four of these modules formed a 256-chip Pod with a peak throughput of approximately 11.5 petaFLOPS.[14][20]

The 2D torus meant that each chip could communicate with its north, south, east, and west neighbors, and wraparound links connected chips on opposite edges of the grid. This was the architectural foundation on which all subsequent Pods were built.

### TPU v3: scaling to 1,024 chips (2018)

TPU v3 retained the 2D torus ICI topology but increased per-link bandwidth and raised per-chip performance to 123 TFLOPS (bf16), 2.7 times v2's 45 TFLOPS. Pods scaled from 256 to 1,024 chips, and the aggregate Pod throughput exceeded 100 petaFLOPS (Google's documentation now lists 126 petaFLOPS in bf16).[14][22] The higher power density of v3 chips required liquid cooling for the first time in Google's TPU program. In the MLPerf v0.6 training round of July 2019, Google's largest ResNet-50, SSD and Transformer submissions used full 1,024-chip TPU v3 Pods, and Google said it set three performance records.[25][26]

### TPU v4: 3D torus and optical switches (2021)

TPU v4 was a major architectural leap. The interconnect moved from a 2D torus to a **3D torus**, where each chip connects to six neighbors along three axes instead of four neighbors in two axes. This lowers the network diameter: in a torus a message only has to travel halfway around each ring, so the worst-case path in a square 2D torus of N chips is about the square root of N hops, against about 1.5 times the cube root of N in a cubic 3D torus.[24]

The most significant innovation was the introduction of **Optical Circuit Switches (OCS)**; the TPU v4 paper describes v4 as the first supercomputer built with a reconfigurable interconnect that enhances performance.[2] Pods scaled to 4,096 chips and delivered 1.1 exaFLOPS (bf16).[2] The TPU v4 system was described in a paper presented at ISCA 2023.[2]

### TPU v5e and v5p: cost and performance tiers (2023)

Google split the fifth generation into two products. [TPU v5e](https://aiwiki.ai/wiki/google_tpu_v5e) was a cost-efficient chip using a 2D torus with Pods of up to 256 chips, aimed at inference and moderate-scale training. TPU v5p was the performance variant, scaling to 8,960 chips in a 3D torus and delivering about 4.1 exaFLOPS of bf16 compute across a full Pod (8,960 chips at 459 TFLOPS each).[9][11] Google states that each v5p Pod "composes together 8,960 chips over our highest-bandwidth inter-chip interconnect (ICI) at 4,800 Gbps/chip in a 3D torus topology."[11] In MLPerf Training, Google entered a 4,096-chip TPU v5e system in round v3.1 (November 2023) and a 6,144-chip TPU v5p system in v4.0 (June 2024).[27][28]

### TPU v6e / Trillium (2024)

[Trillium](https://aiwiki.ai/wiki/google_trillium) doubled ICI bandwidth over v5e and achieved 918 TFLOPS per chip (bf16). Pods scale to 256 chips in a 2D torus.[21] Google reported that 100,000 Trillium chips can be connected within a single Jupiter data center fabric with 13 petabits per second of bisection bandwidth.[20] Google entered Trillium systems of up to 3,072 chips, as a preview, in MLPerf Training v4.1 in November 2024.[29]

### TPU v7 / Ironwood (2025)

Ironwood represents the largest TPU Pod Google offers as of October 2026.[12] A single rack of hosts contains 64 chips arranged as a 4x4x4 "cube" connected in a 3D torus. Multiple cubes are linked through OCS connections: Google offers a 256-chip pod built from four cubes and a 9,216-chip superpod built from 144 cubes, with 42.5 exaFLOPS of FP8 compute.[12][13][16] Each Ironwood chip carries 192 GB of HBM3E and an ICI bandwidth of 9.6 Tb/s (1.2 TB/s) per chip, and a full superpod exposes about 1.77 PB of directly accessible HBM.[12][13][16] The superpod is slightly larger than a power of two so that it holds spare racks: according to ServeTheHome's report of Google's Hot Chips 2025 talk, 8,192 chips would have left no spares.[19] Beyond one superpod, Google links superpods over its Jupiter data-center network into clusters of hundreds of thousands of chips.[17]

### TPU 8t and TPU 8i (announced 2026)

Google's eighth generation, announced in April 2026 and due for general availability later that year, splits Pods by workload. TPU 8t keeps the 3D torus but grows the superpod to 9,600 chips, with about 2 PB of shared HBM, 121 exaFLOPS of FP4 compute and double Ironwood's interchip bandwidth; Google's new Virgo network can link more than 134,000 TPU 8t chips in one non-blocking fabric, and Google says JAX and Pathways can scale a training cluster to more than a million chips.[23][24] [TPU 8i](https://aiwiki.ai/wiki/google_tpu_8i) uses a new hierarchical topology called Boardfly instead of a torus: four fully connected chips form a board, eight boards form a group, and 36 groups linked by optical circuit switches make a pod of up to 1,152 chips (1,024 active), with any chip at most seven hops from any other.[24]

## What are the Pod specifications by generation?

The following table summarizes key Pod-level specifications across TPU generations:

| Generation | Year | Max chips per Pod | Topology | Per-chip bf16 TFLOPS | Per-chip HBM | ICI bandwidth per chip | Pod peak compute |
|---|---|---|---|---|---|---|---|
| TPU v2 | 2017 | 256 | 2D torus | 45 | 16 GB | N/A | 11.5 PFLOPS |
| TPU v3 | 2018 | 1,024 | 2D torus | 123 | 32 GB | N/A | 126 PFLOPS |
| [TPU v4](https://aiwiki.ai/wiki/google_tpu_v4) | 2021 | 4,096 | 3D torus + OCS | 275 | 32 GB | N/A | 1.1 EFLOPS |
| [TPU v5e](https://aiwiki.ai/wiki/google_tpu_v5e) | 2023 | 256 | 2D torus | 197 | 16 GB | 400 GB/s | 50.6 PFLOPS |
| [TPU v5p](https://aiwiki.ai/wiki/google_tpu_v5p) | 2023 | 8,960 | 3D torus + OCS | 459 | 95 GB | 1,200 GB/s | ~4.1 EFLOPS |
| TPU v6e (Trillium) | 2024 | 256 | 2D torus | 918 | 32 GB | 800 GB/s | 234.9 PFLOPS |
| TPU v7 (Ironwood) | 2025 | 9,216 | 3D torus + OCS | 2,307 (bf16) / 4,614 (FP8) | 192 GB | 1,200 GB/s | 42.5 EFLOPS (FP8) |
| [TPU 8t](https://aiwiki.ai/wiki/google_tpu_8t) (announced) | 2026 | 9,600 | 3D torus + OCS | Not published (12,600 FP4) | 216 GB | 2x Ironwood | 121 EFLOPS (FP4) |
| [TPU 8i](https://aiwiki.ai/wiki/google_tpu_8i) (announced) | 2026 | 1,152 | Boardfly | Not published (10,100 FP4) | 288 GB | 2,400 GB/s (19.2 Tb/s) | N/A |

Sources: Google's documentation for v3 through TPU7x and its April 2026 announcements for the eighth generation; v5p's Pod peak is computed from its per-chip figure.[9][13][20][21][22][23][24]

## How does the interconnect architecture work?

### Inter-Chip Interconnect (ICI)

The ICI is a custom high-speed link that directly connects neighboring TPU chips.[7] Unlike [GPU cluster](https://aiwiki.ai/wiki/gpu_cluster)s that route inter-accelerator traffic through PCIe switches, NVLink bridges, or [InfiniBand](https://aiwiki.ai/wiki/infiniband) fabrics, ICI provides a direct, low-latency chip-to-chip path with terabit-per-second bandwidth.[10][15] No host CPU is involved in ICI communication: Google describes ICI as supporting remote direct memory access that lets chips exchange data while bypassing the host CPU.[16]

Key characteristics of ICI:

- **Nearest-neighbor connectivity.** Each chip connects only to its immediate neighbors in the torus. In a 2D torus, each chip has 4 ICI links; in a 3D torus, each chip has 6.[8][21]
- **Multi-hop routing.** Communication between non-adjacent chips is routed through intermediate chips. The torus wraparound links cut the maximum hop count in half compared to a simple mesh.[15]
- **Bandwidth hierarchy.** On TPU v5p, for example, each ICI link carries about 90 GB/s in each direction, and Google lists 1,200 GB/s of bidirectional ICI bandwidth per chip. This is well below the chip's HBM bandwidth (2,765 GB/s) but far above its data center network bandwidth (about 6.25 GB/s per chip).[9][15]

### 2D torus topology

TPU v2, v3, v5e, and v6e use a 2D torus. In this layout, chips are arranged in a rectangular grid with wraparound links connecting each edge to the opposite edge. Each chip connects to four neighbors: north, south, east, and west.

The 2D torus Pods of TPU v5e and v6e are 16x16 (256 chips), while TPU v3 Pods were 32x32 (1,024 chips).[15][21] This topology is simpler to program and sufficient for workloads that can be partitioned along two dimensions, such as data parallelism across one axis and model parallelism across the other.

### 3D torus topology

TPU v4, v5p, and v7 (Ironwood) use a 3D torus, adding a third axis with wraparound links. Each chip connects to six neighbors. The three-dimensional layout provides several advantages over a 2D torus:

- **Lower network diameter.** Because a message only has to travel halfway around each ring, the worst case in a k x k x k torus is about 1.5k hops (1.5 times the cube root of N), against about k hops for a square 2D torus of N = k x k chips. At 4,096 chips, that is about 24 hops for a 16x16x16 torus, compared with about 64 for a 64x64 one. Google used the same arithmetic in 2026 to put an 8x8x16 torus of 1,024 chips at 16 hops.[24]
- **Higher bisection bandwidth.** The 3D torus has more links crossing any bisecting plane, providing greater bandwidth for collective operations like all-reduce.
- **Three axes for parallelism.** The three physical dimensions can be mapped to data parallelism, model (tensor) parallelism, and [pipeline parallelism](https://aiwiki.ai/wiki/pipeline_parallelism) simultaneously.

The basic building block of a 3D torus Pod is a **cube** of 4x4x4 = 64 chips. Cubes are connected to each other via wraparound links, which in TPU v4, v5p and Ironwood pass through Optical Circuit Switches.[2][7]

### Twisted torus

TPU v4 introduced a **twisted torus** variant, where the wraparound links are shifted so that a chip at position (x, y, z) connects not to position (x, y, 0) on the opposite edge but to a position offset by a fixed amount.[2] The twisted torus increases bisection bandwidth by approximately 70% for certain slice shapes (for example, a 4x4x8 twisted torus vs. a standard 4x4x8 torus).[2][8] Because the twist is implemented through OCS routing tables rather than physical rewiring, users can choose between a standard torus and a twisted torus for any given workload; Google's documentation lists twisted topologies for TPU v4 and v5p.[2][9]

### Optical Circuit Switches (OCS)

OCS technology is one of the most distinctive features of TPU v4, v5p and Ironwood Pods.[2][7] An OCS uses arrays of tiny mirrors built with **Micro-Electro-Mechanical Systems (MEMS)** technology to steer optical signals between fiber-optic cables.[2][6] Switching happens in milliseconds, and the switch steers light directly rather than converting the signal back to electrical form.[2][6]

The OCS layer sits between cubes. Within a cube, chips are connected via direct electrical (copper) ICI links in a 3D mesh. The OCS provides the wraparound links that turn this mesh into a torus, and it can dynamically reconfigure which cubes are connected to each other.[7]

Benefits of OCS include:

| Benefit | Description |
|---|---|
| Fault tolerance | If a chip, cable, or OCS port fails, the fabric manager reconfigures optical paths to bypass the fault. Jobs continue on healthy hardware without manual intervention. |
| Flexible partitioning | A single physical Pod can be subdivided into multiple independent slices for different users or workloads. |
| Topology selection | Users can select standard torus, twisted torus, or other topologies through software configuration. |
| Low cost and power | OCS infrastructure accounts for less than 5% of total system cost and less than 3% of total system power, far cheaper and more efficient than electrical switching (e.g., [InfiniBand](https://aiwiki.ai/wiki/infiniband)).[2] |

### Data center network (DCN)

For workloads that span multiple Pods or multiple slices, TPU hosts communicate through Google's data center network.[10] DCN bandwidth per chip is much lower than ICI bandwidth (roughly 6.25 GB/s per chip on v5p, compared with 1,200 GB/s of bidirectional ICI bandwidth), so the XLA compiler and runtime schedule DCN communication carefully to overlap it with computation.[9][10][15]

## What are slices and multislice?

### Slices

A **slice** is a contiguous set of TPU chips within a single Pod, all connected via ICI. Slices are the unit of allocation in Cloud TPU: when a user requests [TPU resource](https://aiwiki.ai/wiki/tpu_resource)s, they receive a slice of a specific topology (for example, a v4-128 slice is a 4x4x4 cube of 64 chips assigned as 128 TensorCores, since each v4 chip has two TensorCores).[7][8]

Slice sizes vary by TPU generation. TPU v4 slice names count TensorCores, so they run from v4-8 (4 chips in a 2x2x1 shape) upward, with v4-4096 being a 2,048-chip 8x16x16 slice; slices above 64 chips come in increments of 64 chips.[8] For TPU v5p, configurations range from 4 chips (2x2x1) up to 6,144 chips (16x16x24), which is the largest schedulable job size; the full v5p Pod has 140 cubes, against the 96 cubes of the largest slice.[9]

### Multislice

Cloud TPU **Multislice** is a scaling technology that allows a single training job to span multiple slices. Chips within each slice communicate via ICI as usual, while chips in different slices exchange data through DCN by routing traffic through host CPUs.[10]

The XLA compiler automatically generates the inter-slice DCN communication code. Developers do not need to write explicit networking logic; the compiler inserts the necessary collective operations and overlaps them with computation to hide latency.[10]

Multislice scaling has been demonstrated at very large scale. In November 2023 Google ran what it believed was the largest publicly disclosed LLM training job by chip count, training models of 16 billion to 128 billion parameters on a cluster of 50,944 TPU v5e chips (199 v5e Pods).[10] Among Multislice's benefits, Google lists: "Train massive models with near-linear scaling performance from single slice to multiple slices with up to tens of thousands of chips."[30] Apple trained its 2024 server foundation model on 8,192 TPU v4 chips provisioned as eight 1,024-chip slices connected over the data center network.[37]

| Feature | Single slice | Multislice |
|---|---|---|
| Communication | ICI only | ICI within slice, DCN between slices |
| Max scale (v5p) | 6,144 chips | 18,432 chips |
| Max scale (v5e) | 256 chips | 50,944 chips demonstrated |
| Latency | Low (ICI) | Higher (DCN adds latency) |
| Programming model | Transparent (SPMD) | Transparent (XLA-managed) |

## How are TPU Pods programmed?

### XLA and GSPMD

The primary compiler for TPU Pods is XLA (Accelerated Linear Algebra), a domain-specific compiler that translates high-level ML operations into optimized TPU machine code. XLA performs whole-program analysis, fusing operations, tiling computations to fit in on-chip memory, and scheduling data transfers to keep the hardware busy.

For distributed workloads on TPU Pods, XLA uses **GSPMD (General-purpose Single Program Multiple Data)**, a partitioning pass that automatically shards a computation across all chips in the Pod. Developers annotate tensors with sharding specifications (for example, "shard this tensor's batch dimension across the first axis of the Pod"), and GSPMD transforms the single-device program into a distributed one, inserting the correct collective communication operations.[5]

This approach means developers can write code as if it will run on a single large device. The compiler handles:

- Partitioning the model and data across chips
- Inserting [all-reduce](https://aiwiki.ai/wiki/all_reduce), all-gather, and reduce-scatter collectives
- Overlapping communication with computation
- Mapping logical tensor dimensions to physical Pod topology axes

### Framework support

TPU Pods are supported by three major ML frameworks, although Ironwood supports only JAX and PyTorch:[13]

| Framework | TPU Pod mechanism | Distributed training approach |
|---|---|---|
| [JAX](https://aiwiki.ai/wiki/jax) | Native XLA, `jit` + `shmap`/`pjit` | GSPMD sharding annotations; write for one device, compiler distributes |
| [TensorFlow](https://aiwiki.ai/wiki/tensorflow) | `tf.distribute.TPUStrategy` | [Data parallelism](https://aiwiki.ai/wiki/data_parallelism) and [model parallelism](https://aiwiki.ai/wiki/model_parallelism) via distribution strategies |
| [PyTorch](https://aiwiki.ai/wiki/pytorch) | PyTorch/XLA with SPMD | XLA-based sharding; FSDP and [tensor parallelism](https://aiwiki.ai/wiki/tensor_parallelism) supported |

[JAX](https://aiwiki.ai/wiki/jax) is the most commonly used framework for TPU Pod workloads at Google. Its functional programming model aligns naturally with XLA's compilation requirements, and JAX transformations like `jit`, `vmap`, `pmap`, and `shmap` map directly to Pod topologies.[5]

### Parallelism strategies

TPU Pods support all standard distributed training strategies:

- **Data parallelism.** The model is replicated across chips, and each chip processes a different mini-batch. Gradients are synchronized via all-reduce over ICI. This is the simplest and most common strategy.
- **Model parallelism ([tensor parallelism](https://aiwiki.ai/wiki/tensor_parallelism)).** Individual layers are split across chips, with each chip holding a slice of the weight tensors. Activations are communicated between chips during the forward and backward passes.
- **Pipeline parallelism.** Different layers of the model are assigned to different groups of chips. Micro-batches flow through the pipeline, with each group processing a different micro-batch at any given time.
- **[Expert parallelism](https://aiwiki.ai/wiki/expert_parallelism).** Used in [Mixture of Experts (MoE)](https://aiwiki.ai/wiki/mixture_of_experts) architectures, where different expert sub-networks reside on different chips and a gating network routes tokens to the appropriate expert.

In practice, large-scale training on TPU Pods combines multiple strategies. For example, PaLM 540B used a combination of data parallelism and model parallelism across 6,144 TPU v4 chips spanning two Pods.[4]

## How do TPU Pods handle hardware failures?

Operating thousands of chips continuously for days or weeks of training requires robust fault-handling mechanisms. At the scale of a TPU v4 Pod, hardware failures are not rare events but routine occurrences.[3]

### Failure rates

A study by Zu et al., presented at NSDI 2024, reported the following average daily failure rates for the components of Google's TPU v4 supercomputers:[3]

| Component | Daily failure rate |
|---|---|
| TPU machines | 0.08% |
| ICI cables | 0.005% |
| Optical circuit switches | 0.04% |

A 4,096-chip TPU v4 Pod has 1,024 four-chip machines, so the machine failure rate works out to just under one machine failure per day; the paper notes that because each supercomputer runs many jobs at once, the number of jobs affected by hardware outages is still significant.[3][4] Without automated recovery, such failures would cause frequent training interruptions.

### Automated recovery

Google's TPU infrastructure uses several techniques to maintain high availability:

- **OCS reconfiguration.** When the fabric manager detects a failed chip or link, it reconfigures the optical switches to route around the fault, connecting healthy cubes into a new slice that excludes the defective hardware. This happens without human intervention.[3]
- **ICI resiliency.** For TPU v4, v5p, and Ironwood, ICI connections can be routed around OCS and optical ICI faults, maintaining slice availability with only temporary performance degradation.[7]
- **Spare capacity.** Pods hold more cubes than the largest schedulable slice (a v5p Pod has 140 cubes, while its largest slice uses 96), and on Ironwood the OCS fabric manager swaps in a designated spare when a cube or link fails.[9][16]
- **Checkpointing.** Training frameworks save model state to persistent storage at regular intervals. When a failure occurs, training restarts from the most recent checkpoint rather than from scratch.
- **In-memory redundancy.** For training Gemini, Google used redundant in-memory copies of model state distributed across replicas. On hardware failure, intact replicas provided the model state for recovery, avoiding the latency of reading checkpoints from storage.[35]

The Zu et al. study reported that TPU v4 supercomputers achieve **99.98% system availability** through these automated mechanisms, with hardware outages affecting approximately 1% of training jobs.[3]

## What models are trained on TPU Pods?

The following table lists major models and systems trained on TPU Pods:

| Model | Year | TPU generation | Pod scale | Notes |
|---|---|---|---|---|
| [BERT](https://aiwiki.ai/wiki/bert) | 2018 | Cloud TPU (version not stated) | 64 chips (BERT-Large); 16 chips (BERT-Base) | Trained in 4 days; pre-training that transformed [NLP](https://aiwiki.ai/wiki/natural_language_processing) |
| [T5](https://aiwiki.ai/wiki/t5) | 2019 | TPU v3 | Slices of 1,024-chip Pods | Text-to-text framework; explored [scaling laws](https://aiwiki.ai/wiki/scaling_laws) |
| [AlphaFold 2](https://aiwiki.ai/wiki/alphafold) | 2020 | TPU v3 | 128 TPU v3 cores (64 chips) | Solved protein structure prediction; won CASP14 |
| [LaMDA](https://aiwiki.ai/wiki/lamda) | 2021 | TPU v3 | 1,024 chips | Conversational model that powered early Google Bard; pre-trained for about 57.7 days |
| [PaLM](https://aiwiki.ai/wiki/palm) | 2022 | [TPU v4](https://aiwiki.ai/wiki/google_tpu_v4) | 6,144 chips (2 Pods) | 540B parameters; first large-scale use of Pathways system; 57.8% hardware FLOPS utilization |
| [Gemini](https://aiwiki.ai/wiki/gemini) | 2023 | [TPU v4](https://aiwiki.ai/wiki/google_tpu_v4) and v5e | Multi-Pod, across data centers | [Google](https://aiwiki.ai/wiki/google)'s flagship [multimodal model](https://aiwiki.ai/wiki/multimodal_model) family |
| [Gemma](https://aiwiki.ai/wiki/gemma) | 2024 | [TPU v5e](https://aiwiki.ai/wiki/google_tpu_v5e) | 4,096 chips (7B, 16 Pods); 512 chips (2B) | Open-weights model family |
| Apple Foundation Models | 2024 | TPU v4 and [TPU v5p](https://aiwiki.ai/wiki/google_tpu_v5p) | 8,192 v4 chips (server model); 2,048 v5p chips (on-device model) | Models behind Apple Intelligence |

Sources: the model papers and reports listed in the references.[4][31][32][33][34][35][36][37]

PaLM is a particularly instructive example of TPU Pod usage. The 540B-parameter model was trained across two TPU v4 Pods, each with 3,072 chips (for a total of 6,144 chips), using the Pathways system to coordinate computation across Pods. The training achieved 57.8% hardware [FLOPS](https://aiwiki.ai/wiki/flops) utilization, the highest figure reported for LLM training at that scale at the time of publication.[4]

## How do TPU Pods differ from GPU clusters?

The primary alternative to TPU Pods for large-scale ML training is [NVIDIA](https://aiwiki.ai/wiki/nvidia) GPU clusters connected via NVLink and InfiniBand. The two approaches differ in several respects:

| Aspect | TPU Pod | [GPU](https://aiwiki.ai/wiki/gpu) cluster ([NVIDIA](https://aiwiki.ai/wiki/nvidia)) |
|---|---|---|
| Intra-node interconnect | ICI (custom torus, direct chip-to-chip) | NVLink + NVSwitch (within DGX node) |
| Inter-node interconnect | ICI continues across the Pod via OCS | [InfiniBand](https://aiwiki.ai/wiki/infiniband) / RoCE between nodes |
| Topology | 2D or 3D torus | Fat-tree (typically via [InfiniBand](https://aiwiki.ai/wiki/infiniband) switches) |
| Programming model | XLA/GSPMD (compiler-driven sharding) | [CUDA](https://aiwiki.ai/wiki/cuda) / NCCL (explicit collectives) |
| Availability | Mainly [Google Cloud](https://aiwiki.ai/wiki/google_cloud); TPU systems also delivered to some customer data centers since 2026 | Available for purchase; all major clouds |
| Max single-system scale | 9,216 chips (Ironwood superpod); 9,600 announced for TPU 8t | 72 GPUs per NVLink domain (GB200 NVL72); larger via [InfiniBand](https://aiwiki.ai/wiki/infiniband) or Ethernet |
| Software ecosystem | [JAX](https://aiwiki.ai/wiki/jax), [TensorFlow](https://aiwiki.ai/wiki/tensorflow), [PyTorch](https://aiwiki.ai/wiki/pytorch)/XLA | [CUDA](https://aiwiki.ai/wiki/cuda) ecosystem (broad third-party support) |

Sources for the scale and availability rows: Google, NVIDIA and Alphabet.[12][23][41][43]

TPU Pods have a structural advantage in that ICI provides a uniform, high-bandwidth fabric across the entire Pod without the bandwidth bottleneck that occurs between nodes in GPU clusters.[15] In a GPU cluster, NVLink provides very high bandwidth within a single multi-GPU node (e.g., 900 GB/s per GPU in DGX H100), but communication between nodes drops to InfiniBand speeds (typically 400 Gb/s per port). In a TPU Pod, every chip-to-chip link uses ICI regardless of physical distance within the Pod.

GPU clusters, on the other hand, benefit from a much larger and more mature software ecosystem, broader availability across cloud providers, and the ability to be purchased as on-premises hardware.

## How do you access TPU Pods on Google Cloud?

### Pricing

Cloud TPU Pods are billed per chip-hour, and prices vary by generation, region, and commitment level. In October 2026 Google's pricing page listed on-demand rates of $1.20 per chip-hour for TPU v5e, $4.20 for TPU v5p, $2.70 for Trillium (us-east1) and $12.00 for Ironwood (us-central1); one-year and three-year commitments cut those rates by about 30% and 55% respectively.[38] The [Cloud TPU](https://aiwiki.ai/wiki/cloud_tpu) article lists the full, dated price history.

Google also offers **Spot** capacity at variable discounts for fault-tolerant workloads, Dynamic Workload Scheduler options (Flex-start and calendar-mode reservations), and **queued resources** for users who can wait for availability.[20][42]

### Regional availability

TPU Pods are available in select [Google Cloud](https://aiwiki.ai/wiki/google_cloud) regions, and not every generation is offered everywhere. In October 2026 Google's zone list showed Ironwood (TPU7x) only in us-central1, v5p in us-central1, us-east5 and europe-west4, and Trillium in zones across North America, Europe, Asia and South America.[39] Some capacity requires manual approval: Google grants no default quota for TPU v4 in us-central2-b, so all requests there are reviewed by Google.[8]

### TPU Research Cloud (TRC)

Google's **TPU Research Cloud** program provides free Cloud TPU access to researchers. In exchange, participants are expected to share their work through peer-reviewed publications, open-source code, blog posts, or other means.[40]

## See also

- [Tensor Processing Unit (TPU)](https://aiwiki.ai/wiki/tpu)
- [Cloud TPU](https://aiwiki.ai/wiki/cloud_tpu)
- [TPU Ironwood](https://aiwiki.ai/wiki/tpu_ironwood)
- [TPU Board](https://aiwiki.ai/wiki/tpu_board)

## References

1. Jouppi, N. P., et al. "In-Datacenter Performance Analysis of a Tensor Processing Unit." *Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA)*, 2017. [arXiv:1704.04760](https://arxiv.org/abs/1704.04760)
2. Jouppi, N. P., et al. "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings." *Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA)*, 2023. [arXiv:2304.01433](https://arxiv.org/abs/2304.01433)
3. Zu, Y., et al. "Resiliency at Scale: Managing Google's TPUv4 Machine Learning Supercomputer." *Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI)*, 2024. [usenix.org/conference/nsdi24/presentation/zu](https://www.usenix.org/conference/nsdi24/presentation/zu)
4. Chowdhery, A., et al. "PaLM: Scaling Language Modeling with Pathways." *arXiv preprint*, 2022. [arXiv:2204.02311](https://arxiv.org/abs/2204.02311)
5. Yoo, J., et al. "Scalable Training of Language Models using JAX pjit and TPUv4." *arXiv preprint*, 2022. [arXiv:2204.06514](https://arxiv.org/abs/2204.06514)
6. Patel, D., et al. "Lightwave Fabrics: At-Scale Optical Circuit Switching for Datacenter and Machine Learning Systems." *Proceedings of the ACM SIGCOMM Conference*, 2023. [dl.acm.org/doi/10.1145/3603269.3604836](https://dl.acm.org/doi/10.1145/3603269.3604836)
7. Google Cloud. "TPU system architecture." *Cloud TPU Documentation*. [docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm](https://docs.cloud.google.com/tpu/docs/system-architecture-tpu-vm)
8. Google Cloud. "TPU v4 documentation." *Cloud TPU Documentation*. [docs.cloud.google.com/tpu/docs/v4](https://docs.cloud.google.com/tpu/docs/v4)
9. Google Cloud. "TPU v5p documentation." *Cloud TPU Documentation*. [docs.cloud.google.com/tpu/docs/v5p](https://docs.cloud.google.com/tpu/docs/v5p)
10. Google Cloud. "Cloud TPU Multislice overview" and "The world's largest distributed LLM training job on TPU v5e." *Cloud TPU Documentation and Google Cloud Blog*. [docs.cloud.google.com/tpu/docs/multislice-introduction](https://docs.cloud.google.com/tpu/docs/multislice-introduction), [cloud.google.com/blog/products/compute/the-worlds-largest-distributed-llm-training-job-on-tpu-v5e](https://cloud.google.com/blog/products/compute/the-worlds-largest-distributed-llm-training-job-on-tpu-v5e)
11. Google Cloud. "Introducing Cloud TPU v5p and AI Hypercomputer." *Google Cloud Blog*, 2023. [cloud.google.com/blog/products/ai-machine-learning/introducing-cloud-tpu-v5p-and-ai-hypercomputer](https://cloud.google.com/blog/products/ai-machine-learning/introducing-cloud-tpu-v5p-and-ai-hypercomputer)
12. Google. "Ironwood: The first Google TPU for the age of inference." *The Keyword*, 2025. [blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/ironwood-tpu-age-of-inference](https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/ironwood-tpu-age-of-inference/)
13. Google Cloud. "TPU7x (Ironwood) documentation." *Cloud TPU Documentation*. [docs.cloud.google.com/tpu/docs/tpu7x](https://docs.cloud.google.com/tpu/docs/tpu7x)
14. Wikipedia. "Tensor Processing Unit." [en.wikipedia.org/wiki/Tensor_Processing_Unit](https://en.wikipedia.org/wiki/Tensor_Processing_Unit)
15. Scaling Book (JAX). "How to Think About TPUs." [jax-ml.github.io/scaling-book/tpus](https://jax-ml.github.io/scaling-book/tpus/)
16. Google Cloud Blog. "Inside the Ironwood TPU codesigned AI stack." Google Cloud, 2025-11-06. https://cloud.google.com/blog/products/compute/inside-the-ironwood-tpu-codesigned-ai-stack
17. Google Cloud Blog. "Ironwood TPUs and new Axion-based VMs for your AI workloads." Google Cloud, 2025-11-07. https://cloud.google.com/blog/products/compute/ironwood-tpus-and-new-axion-based-vms-for-your-ai-workloads
18. The Register. "Google's 7th-gen Ironwood TPUs promise 42 AI exaFLOPS pods." The Register, 2025-04-10. https://www.theregister.com/2025/04/10/googles_7thgen_ironwood_tpus_debut/
19. ServeTheHome. "Google Ironwood TPU Swings for Reasoning Model Leadership at Hot Chips 2025." ServeTheHome, 2025-08-27. https://www.servethehome.com/google-ironwood-tpu-swings-for-reasoning-model-leadership-at-hot-chips-2025/
20. Google Cloud Documentation. "Cloud TPU release notes." Google Cloud, 2026-10-13. https://docs.cloud.google.com/tpu/docs/release-notes
21. Google Cloud Documentation. "TPU v6e." Google Cloud, 2026-10-13. https://docs.cloud.google.com/tpu/docs/v6e
22. Google Cloud Documentation. "TPU v3." Google Cloud, 2026-10-13. https://docs.cloud.google.com/tpu/docs/v3
23. Google Blog. "Our eighth generation TPUs: two chips for the agentic era." Google, 2026-04-22. https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/eighth-generation-tpu-agentic-era/
24. Google Cloud Blog. "TPU 8t and TPU 8i technical deep dive." Google Cloud, 2026-04-23. https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive
25. MLCommons. "MLPerf Training v0.6 results: Google systems." MLCommons, 2019-07-10. https://github.com/mlcommons/training_results_v0.6/tree/master/Google/systems
26. Google Cloud Blog. "Cloud TPU Pods break AI training records." Google Cloud, 2019-07-11. https://cloud.google.com/blog/products/ai-machine-learning/cloud-tpu-pods-break-ai-training-records
27. MLCommons. "MLPerf Training v3.1 results: Google systems." MLCommons, 2023-11-07. https://github.com/mlcommons/training_results_v3.1/tree/main/Google/systems
28. MLCommons. "MLPerf Training v4.0 results: Google systems." MLCommons, 2024-06-11. https://github.com/mlcommons/training_results_v4.0/tree/main/Google/systems
29. MLCommons. "MLPerf Training v4.1 results: Google systems." MLCommons, 2024-11-13. https://github.com/mlcommons/training_results_v4.1/tree/main/Google/systems
30. Google Cloud Blog. "How to scale AI training to up to tens of thousands of Cloud TPU chips with Multislice." Google Cloud, 2023-09-01. https://cloud.google.com/blog/products/compute/using-cloud-tpu-multislice-to-scale-ai-workloads
31. Devlin, J., et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." arXiv, 2018-10-11. https://arxiv.org/abs/1810.04805
32. Raffel, C., et al. "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." arXiv, 2019-10-23. https://arxiv.org/abs/1910.10683
33. Jumper, J., et al. "Highly accurate protein structure prediction with AlphaFold." Nature, 2021-07-15. https://www.nature.com/articles/s41586-021-03819-2
34. Thoppilan, R., et al. "LaMDA: Language Models for Dialog Applications." arXiv, 2022-01-20. https://arxiv.org/abs/2201.08239
35. Gemini Team, Google. "Gemini: A Family of Highly Capable Multimodal Models." arXiv, 2023-12-19. https://arxiv.org/abs/2312.11805
36. Gemma Team, Google DeepMind. "Gemma: Open Models Based on Gemini Research and Technology." arXiv, 2024-03-13. https://arxiv.org/abs/2403.08295
37. Apple. "Apple Intelligence Foundation Language Models." arXiv, 2024-07-29. https://arxiv.org/abs/2407.21075
38. Google Cloud. "Cloud TPU pricing." Google Cloud, 2026-10-16. https://cloud.google.com/tpu/pricing
39. Google Cloud Documentation. "TPU regions and zones." Google Cloud, 2026-10-13. https://docs.cloud.google.com/tpu/docs/regions-zones
40. Google Research. "TPU Research Cloud." Google, 2026-10-16. https://sites.research.google/trc/about/
41. NVIDIA. "NVIDIA GB200 NVL72." NVIDIA, 2026-10-16. https://www.nvidia.com/en-us/data-center/gb200-nvl72/
42. Google Cloud. "Spot VMs pricing." Google Cloud, 2026-10-16. https://cloud.google.com/spot-vms/pricing
43. Alphabet. "Q2 2026 Earnings Call Transcript." Alphabet Investor Relations, 2026-07-22. https://s206.q4cdn.com/479360582/files/doc_events/2026/Jul/22/2026_Q2_Earnings_Transcript.pdf