tpu

A TPU — Tensor Processing Unit — is Google's custom-built chip for neural network math, and architecturally it takes a genuinely different approach from the GPU covered under gpu rather than just being a competing brand of the same idea. Jouppi and colleagues' 2017 paper "In-Datacenter Performance Analysis of a Tensor Processing Unit" revealed that Google had already been running the first TPU internally in its datacenters since 2015, purpose-built for inference workloads rather than the general-purpose parallel computing a GPU is designed for — and the core of that purpose-built design is a structure called a systolic array, which handles matrix multiplication in a fundamentally different way than the GPU's thousands of independent cores do.

Input matrix enters one edge
of a grid of multiply-accumulate (MAC) units
Data flows through the grid
each MAC unit multiplies and passes the partial sum onward
Partial sums accumulate in motion
no round-trip to memory between steps — the array IS the computation
Completed result emerges at the far edge
a full matrix multiply, computed almost entirely in the array itself

The systolic array's defining trick is that data almost never has to leave the array and round-trip to memory mid-computation, which is exactly the opposite of how a GPU's many independent cores operate. Each cell in the grid holds one value, multiplies it against an incoming value, adds the result to a running sum, and passes that sum to the next cell in line — the computation moves through the hardware the way blood moves through a heart, which is literally where the name "systolic" comes from — so a large matrix multiplication, the single most common operation in a neural network, gets executed with far less memory traffic than a general-purpose core array would need to shuffle the same data through repeatedly. A GPU's SIMT cores, by contrast, each independently read, multiply, and write back to memory, which is more flexible for a wider range of computations but spends more memory bandwidth getting there.

Data Flows Through the Array Instead of Round-Tripping to Memory a systolic array — the structure underneath every TPU since the 2017 paper input → partial sums flow right too Result emerges here no intermediate memory round-trip A GPU's cores each independently read, compute, and write back to memory — flexible for many kinds of work, but spends bandwidth getting there every time. The systolic array trades that flexibility for far less memory traffic on one specific operation. Same goal — fast matrix multiply — two structurally different hardware answers.

Each TPU generation traded the first one's inference-only focus for broader training capability, and the real version history traces that shift. The original TPU described in the 2017 paper handled only inference; TPU v2, released that same year, added training capability; TPU v3 in 2018 moved to liquid cooling to handle the resulting heat; TPU v4, released in 2021, scaled up to pods of thousands of interconnected chips working as one system; and the v5 generation, released in 2023, split into two specialized variants — v5e tuned for cost-efficient inference, and v5p tuned for training throughput — reflecting that inference and training workloads had diverged enough to be worth separate hardware optimized for each.

TPUs require a compiler step a GPU workload doesn't strictly need, because the systolic array's fixed structure has to be fed computation in exactly the shape it expects. XLA (Accelerated Linear Algebra), Google's compiler for TPU workloads, translates a model's computation graph into the specific tiled matrix operations the systolic array hardware can actually execute efficiently — which is why TPU software support centers on JAX and TensorFlow, both built around XLA from the ground up, while PyTorch's TPU support runs through a separate compatibility layer rather than being native the way CUDA is to NVIDIA GPUs. This narrower, more compiler-dependent software ecosystem is the direct cost of the systolic array's efficiency gain.

The GPU-versus-TPU choice comes down to the same trade-off as any specialized-versus-general-purpose hardware decision: a systolic array wins decisively on the one operation — large matrix multiplication — that dominates neural network training and inference, while a GPU's more flexible core array handles a wider range of workloads reasonably well without being optimized to the same degree for any single one, which is exactly why TPUs exist inside Google's own infrastructure specifically rather than as a general-purpose alternative the rest of the industry builds around.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account