Tensor Processing Unit (TPU)

A Tensor Processing Unit (TPU) is a family of custom application-specific integrated circuits developed by Google to accelerate machine-learning computation. The original TPU was an inference accelerator deployed in Google data centers in 2015. Its 256 by 256 matrix unit contained 65,536 8-bit multiply-accumulators and delivered a peak 92 trillion operations per second (TOPS) ^[1]. The family later expanded into training and inference systems offered through Google Cloud. TPU7x, the first Ironwood system, became generally available on March 31, 2026 ^[2]. Google announced the eighth generation, TPU 8t and TPU 8i, in April 2026, but said they would become available to cloud customers later; they were therefore announced rather than generally available products at the research cutoff of July 28, 2026 ^[3].

Scope and terminology

The name TPU covers several related levels of a system. A TPU chip contains one or more compute cores, and cloud documentation sometimes exposes multiple logical devices per physical chip. A slice is a provisioned group of chips inside one pod, while a pod is the larger set connected by Google's inter-chip interconnect (ICI). Multislice jobs can span more than one slice or pod through a data-center network. Counts stated in cores, devices, chips, slices, and pods are therefore not interchangeable ^[2].

This article primarily concerns Google's data-center TPU family. The Coral Edge TPU is a separate, small inference accelerator discussed near the end of the article. Google's Pixel Tensor system-on-chips and Nvidia products also called Tensor Cores are distinct products. Within a data-center TPU, Google uses TensorCore for a compute block that contains matrix-multiply units (MXUs), a vector unit, and a scalar unit; that name does not denote Nvidia's tensor-core architecture ^[4].

TPUs are AI accelerators, not general-purpose replacements for a CPU or GPU. Their design concentrates silicon and memory bandwidth on the matrix, vector, and collective operations common in neural networks. This specialization can produce high throughput on suitable training and inference workloads, but it also makes results more dependent on model shape, numerical format, compiler quality, communication, and memory behavior than a single peak-operations figure suggests ^[1][4].

History

Google began the TPU project after an internal 2013 estimate found that a modest increase in neural-network voice search could require roughly twice the company's data-center compute capacity. The first design was completed on an unusually short schedule and deployed in 2015 as a PCIe-attached inference coprocessor. Google announced it publicly at Google I/O in May 2016 ^[5]. The subsequent ISCA paper evaluated six production neural-network applications and compared the chip with the Intel Haswell CPU and Nvidia K80 GPU used in the same period. Across those particular 2015 workloads, the TPU was 15 to 30 times faster and delivered 30 to 80 times as many operations per watt. Those historical results should not be read as a general comparison with later CPUs or GPUs ^[1].

TPU v2, announced in 2017, added floating-point support for both training and inference, high-bandwidth memory, and a direct chip-to-chip network. TPU v3 increased compute, memory, and system scale, and used liquid cooling. The published v2 and v3 systems connected 256 and 1,024 chips, respectively, in two-dimensional torus networks ^[6]. Google opened Cloud TPU beta access in February 2018, allowing customers to attach TPUs to cloud virtual machines rather than buy the chips ^[7].

Later systems increased both per-chip capability and pod scale. TPU v4 introduced a three-dimensional network with optical circuit switches. The fifth generation split into v5e, oriented toward economical training and serving, and v5p, oriented toward large training systems. TPU v6e, named Trillium, returned to a 256-chip pod design while increasing per-chip compute. TPU7x, the first Ironwood configuration, became generally available in March 2026. In April 2026, Google announced separate eighth-generation designs for training and inference instead of one chip serving both roles ^[2][3].

Architecture

Systolic matrix processing

The central compute structure is an MXU built from a systolic array. Data and partial sums move through a grid of multiply-accumulate cells in a regular schedule. Reusing values inside the grid reduces repeated reads from external memory during matrix multiplication. TPU v1 used a 256 by 256 INT8 array. Data-center generations before v6e used 128 by 128 MXUs for their main floating-point path, while v6e and TPU7x use 256 by 256 MXUs ^[1][8].

A TensorCore combines one or more MXUs with a vector unit for operations such as activations and softmax, plus a scalar unit for control and address calculation. TPU software stages data through a hierarchy that includes host memory, per-chip high-bandwidth memory, and software-managed on-chip vector memory. The compiler must arrange computation and movement so that the MXUs remain occupied. Small matrices, irregular control flow, excessive recompilation, or data pipelines that cannot feed the device can leave peak arithmetic capacity unused ^[8].

Numerical formats

TPU v1 performed 8-bit integer inference. TPU v2 introduced bfloat16 for training. Bfloat16 has one sign bit, eight exponent bits, and seven fraction bits, so it retains the exponent range of IEEE binary32 while using half as many bits. TPU multiplication commonly accepts bfloat16 inputs and accumulates into 32-bit floating point. Later generations added lower-precision paths, including FP8 on Ironwood and FP4 on the announced eighth generation ^[8][9].

Peak figures at different precisions are not directly comparable. For example, TPU7x is rated at 2,307 BF16 teraflops and 4,614 FP8 teraflops per chip, while TPU 8t was announced at 12.6 FP4 petaflops. A model must be able to use the stated format without unacceptable numerical loss, and measured throughput also depends on memory traffic, interconnect traffic, operation mix, batch size, and compiler-generated code ^[3][8].

Sparse and collective engines

Dense systolic arrays are inefficient for some embedding lookups and other irregular, memory-heavy operations used in recommendation systems. Google's TPU v4 paper publicly described the SparseCore, a dedicated dataflow processor for embedding workloads. In Google's evaluation, it used about 5 percent of die area and power and produced five to seven times the embedding performance of the preceding approach on the tested workloads ^[10]. Later TPUs continued the feature, with four SparseCores per TPU7x chip ^[11].

The announced TPU 8t combines SparseCore with an LLM Decoder Engine and is oriented toward large-scale pre-training. TPU 8i instead adds a Collectives Acceleration Engine and uses a hierarchical "Boardfly" network intended for communication-heavy serving, reasoning, and mixture-of-experts workloads. These are vendor-stated design roles, not proof that either system is faster for every workload ^[12].

Generations

The following table uses per-chip peak arithmetic and memory specifications from Google documentation and published architecture papers. "System size" means the pod or published supercomputer size identified by the cited source, not necessarily the largest slice that a cloud customer can schedule. Arithmetic figures retain their stated numerical format.

Scroll sideways for more →

The v1 values come from its 2017 architecture paper ^[1]. The v2 row describes the 256-chip system published in 2020 ^[6]; current Cloud TPU documentation separately describes a 512-chip v2 slice, illustrating why chip, slice, and historical system counts must be labeled ^[13]. Current Google documentation provides the v3 through v6e specifications ^{[14][15][16][17][18]}, while the TPU7x documentation provides the Ironwood row ^[11].

TPU v4 and optical switching

TPU v4 changed the scale-out design from the fixed two-dimensional networks used by v2 and v3. Chips have direct neighbors in three dimensions, forming a 3D mesh. Supported slice shapes can close the dimensions into a 3D torus ^[15]. Optical circuit switches reconfigure links between groups of chips. This lets the system assemble different slice topologies, route around unavailable components, and improve fleet utilization without placing optical transceivers in the packet data path. The v4 paper reports that the optical components contributed less than 5 percent of system cost and less than 3 percent of system power ^[10].

TPU v5, Trillium, and Ironwood

TPU v5e and v6e use 256-chip, two-dimensional pods. V5e provides 197 BF16 teraflops and 16 GB of HBM per chip; v6e raises those values to 918 teraflops and 32 GB. V5p instead uses a larger three-dimensional system: its physical pod contains 8,960 chips, although the largest schedulable single slice is 6,144 chips ^[16][17][18].

TPU7x is a dual-chiplet device. Each physical chip contains two TensorCores, four SparseCores, and 192 GiB of HBM. JAX exposes the two chiplets as separate devices, each with its own memory space. A full 9,216-chip pod has a stated peak of about 21.3 exaflops at BF16 or 42.5 exaflops at FP8, obtained by multiplying the documented per-chip ratings by pod size. Google supports JAX and PyTorch on TPU7x but not TensorFlow ^[11].

Announced eighth generation

Google announced TPU 8t and TPU 8i in April 2026 and said they would be available to cloud customers soon ^[3]. Because no general-availability event had appeared in the Cloud TPU release notes by July 28, they should not be described as shipping Cloud TPU generations at that cutoff ^[2].

Scroll sideways for more →

Google stated that one TPU 8t superpod would contain 9,600 chips, 2 PB of shared memory, and 121 exaflops of FP4 peak compute. The TPU 8i design was announced with 19.2 Tb/s of bidirectional ICI bandwidth. Its Boardfly topology groups boards hierarchically and uses optical switches to reduce the maximum path length for communication patterns such as all-to-all collectives ^[3][12]. These are announced specifications and should remain labeled as such until production documentation establishes available configurations.

Scaling from chips to pods

ICI connects neighboring chips without routing their traffic through host CPUs. A two-dimensional or three-dimensional topology gives each chip a defined set of direct neighbors. Collective operations such as all-reduce then combine gradients or exchange model state across the slice. Topology matters because a workload that communicates only with nearby partitions stresses the network differently from one that frequently exchanges data among all partitions ^[6][8].

A slice is a contiguous provisioned subset of one TPU pod. Smaller jobs can use part of a pod; larger jobs can use a full pod or multislice execution. V4 and v5p also support twisted torus shapes that increase bisection bandwidth for some configurations. TPU7x uses optical reconfiguration at 9,216-chip pod scale, while TPU 8i's announced Boardfly design prioritizes smaller network diameter for all-to-all communication ^{[15][17][11][12]}.

Scaling efficiency is not guaranteed by chip count. Distributed large language model training partitions parameters, optimizer state, activations, and batches across devices. Performance can be limited by collective communication, high-bandwidth-memory capacity, input delivery, checkpoints, stragglers, or work lost to faults. The 2026 retrospective on five TPU supercomputer generations describes hardware, compiler, resilience, scheduling, and power infrastructure as co-designed parts of the system, rather than treating the chip alone as the product ^[4].

Software stack

TPU programs are normally expressed in JAX, TensorFlow, or PyTorch, then compiled for the device. XLA accepts framework graphs represented through StableHLO, applies graph and buffer optimizations, and invokes a target-specific backend to generate executable code ^[19]. PJRT provides a runtime and device interface. JAX uses XLA for compilation and PJRT for device and memory management ^[20].

Compilation lets XLA fuse operations, choose layouts, allocate buffers, and map computation across devices. It also creates constraints. New input shapes can trigger compilation, unsupported operations can fall back or fail, and code with highly dynamic shapes or Python-side work inside the step loop may need restructuring. Profiling therefore has to distinguish device compute, memory bandwidth, collective communication, input stalls, and compilation time ^[8][19].

JAX is the main research and systems environment described in Google's 2026 TPU retrospective. Pallas lets developers write lower-level kernels in Python for TPU and other accelerators, exposing more control over memory movement and tiling ^[4][20]. PyTorch support has historically used PyTorch/XLA. The project announced a successor called TorchTPU in April 2026, but its repository said it would replace PyTorch/XLA when publicly available, so the announcement should not be presented as an already completed migration ^[21]. Framework support also varies by hardware generation: TPU7x supports JAX and PyTorch, while its production documentation explicitly excludes TensorFlow ^[11].

Access and representative workloads

External users access data-center TPUs as cloud resources rather than buying accelerator cards. As of June 2026, Compute Engine APIs could create and manage TPU virtual machines and slices, and Google Kubernetes Engine could also manage supported TPU generations. TPU VM architecture gives each host direct access to attached accelerators, while slices provide the ICI-connected scale-up domain ^[2][8].

The TPU Research Cloud provides accepted researchers with free access to a cluster of more than 1,000 Cloud TPU devices, with an expectation that results, code, or other research outputs will be shared publicly ^[22]. Availability, quota, supported regions, schedulable shapes, and framework versions differ by generation, so production planning should use the current configuration and release documentation rather than a static price or region table.

TPUs have supported both Google production services and research systems. The first TPU evaluated search ranking, image, translation, and game-playing inference workloads ^[1]. The PaLM paper reports training its 540-billion-parameter model on 6,144 TPU v4 chips across two pods using the Pathways system ^[23]. Such examples establish that TPUs can run large models at scale, but they do not imply that the same configuration, utilization, or cost transfers to another model.

Performance interpretation and limitations

A sound TPU comparison specifies at least the hardware generation, device count, numerical format, model, batch or sequence shape, software version, and whether the reported number is peak arithmetic, measured application throughput, latency, energy, or cost. Comparing TPU FP8 or FP4 peak values with GPU BF16 values, or comparing a full TPU pod with one accelerator, is not meaningful without normalization. Vendor price-performance claims can also depend on cloud discounts, utilization assumptions, and selected workloads.

Important practical limits include:

• Specialization: MXUs are efficient for large regular matrix operations, but branch-heavy, sparse, very small, or unsupported operations can underuse the device.
• Memory and communication: HBM capacity, HBM bandwidth, ICI topology, and data-center networking can matter as much as arithmetic peak.
• Compilation: shape changes and unsupported operations can increase compile time or require code changes.
• Framework variation: supported features and recommended software differ by TPU generation; TPU7x, for example, does not support TensorFlow ^[11].
• Procurement and portability: data-center TPUs are consumed through Google Cloud or Google-operated infrastructure, so users cannot deploy the same accelerator on premises or through another cloud provider.
• Availability: a named architecture may be in preview, generally available, limited to certain zones, or only announced. TPU7x moved from preview in November 2025 to general availability in March 2026, while TPU 8t and TPU 8i remained announced at the cutoff ^[2][3].

The original 15 to 30 times performance and 30 to 80 times performance-per-watt result remains historically important, but it applies to TPU v1, six Google production inference workloads, and the CPU and GPU baselines of 2015 ^[1]. It is not evidence for a universal advantage over current GPUs.

Edge TPU

The Edge TPU is a separate inference ASIC used in Coral products. The Coral USB Accelerator contains an Edge TPU rated at 4 TOPS with INT8 models and a stated efficiency of 2 TOPS per watt. It accepts TensorFlow Lite models compiled for the Edge TPU and connects to a host over USB ^[24]. These specifications do not describe a Cloud TPU, and the Edge TPU does not share the pod-scale ICI system used by data-center generations.

The distinction matters when discussing deployment. A Coral accelerator is a small device for local inference under tight power and connectivity constraints. A Cloud TPU is a hosted data-center accelerator designed to scale from individual chips to networked slices and pods. Claims about model support, numerical formats, memory, performance, or availability should identify which product family they concern.

See also

• AI chip
• Deep learning
• Distributed training
• High Bandwidth Memory
• Trillium
• Google TPU 8i

References

1. ^Jouppi, N. P., et al. "In-Datacenter Performance Analysis of a Tensor Processing Unit." *Proceedings of the 44th Annual International Symposium on Computer Architecture*, 2017. arxiv.org/...1704.04760
2. ^Google Cloud. "Cloud TPU release notes." Updated through July 28, 2026. docs.cloud.google.com/...release-notes
3. ^Google Cloud. "What's next in Google AI infrastructure: Scaling for the agentic era." April 23, 2026. cloud.google.com/...ai-infrastructure-at-next26
4. ^Jouppi, N. P., et al. "Google's Training Supercomputers from TPU v2 to Ironwood: Architectural Stability, Scale, Resilience, Power Efficiency, and Sustainability Across Five Generations." arXiv:2606.15870, 2026. arxiv.org/...2606.15870
5. ^Google Cloud. "An in-depth look at Google's first Tensor Processing Unit (TPU)." May 12, 2017. cloud.google.com/...rst-tensor-processing-unit-tpu
6. ^Jouppi, N. P., et al. "A Domain-Specific Supercomputer for Training Deep Neural Networks." *Communications of the ACM*, volume 63, issue 7, 2020. doi.org/...3360307
7. ^Google Cloud. "Cloud TPUs: machine learning accelerators now available in beta." February 12, 2018. cloud.google.com/...lerators-now-available-in-beta
8. ^Google Cloud. "TPU architecture." Updated July 17, 2026. docs.cloud.google.com/...system-architecture-tpu-vm
9. ^Kalamkar, D., et al. "A Study of BFLOAT16 for Deep Learning Training." arXiv:1905.12322, 2019. arxiv.org/...1905.12322
10. ^Jouppi, N. P., et al. "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings." *Proceedings of the 50th Annual International Symposium on Computer Architecture*, 2023. arxiv.org/...2304.01433
11. ^Google Cloud. "TPU7x (Ironwood)." docs.cloud.google.com/...tpu7x
12. ^Google Cloud. "TPU 8t and TPU 8i technical deep dive." April 22, 2026. cloud.google.com/...and-tpu-8i-technical-deep-dive
13. ^Google Cloud. "TPU v2." Updated July 22, 2026. docs.cloud.google.com/...v2
14. ^Google Cloud. "TPU v3." Updated July 22, 2026. docs.cloud.google.com/...v3
15. ^Google Cloud. "TPU v4." Updated July 22, 2026. docs.cloud.google.com/...v4
16. ^Google Cloud. "TPU v5e." docs.cloud.google.com/...v5e
17. ^Google Cloud. "TPU v5p." docs.cloud.google.com/...v5p
18. ^Google Cloud. "TPU v6e." docs.cloud.google.com/...v6e
19. ^OpenXLA Project. "XLA architecture." openxla.org/...architecture
20. ^JAX project. "About the project." docs.jax.dev/...about
21. ^PyTorch/XLA project. "PyTorch/XLA." GitHub repository, accessed July 28, 2026. github.com/...xla
22. ^Google. "TPU Research Cloud." sites.research.google/trc
23. ^Chowdhery, A., et al. "PaLM: Scaling Language Modeling with Pathways." arXiv:2204.02311, 2022. arxiv.org/...2204.02311
24. ^Google. "Coral USB Accelerator Datasheet." Version 1.4. gweb-coral-full.uc.r.appspot.com/...-datasheet.pdf