ICI, OCS, slices and pods

TPUs connect chip to chip, with no switch in the data path. Each chip has a small number of ICI (inter-chip interconnect) links to its nearest neighbours, and messages for distant chips pass through the chips in between [SB2]. Everything in this chapter follows from that choice: how slices are shaped, why wraparound matters, how far a pod scales, and why TPU 8i changed it.

Converting the bandwidth numbers

Google's documentation quotes ICI bandwidth per chip, summed over all links and both directions. Collective cost formulas need bandwidth per link and per direction. The conversion is one division by the link count and one by two:

Docs: [D-v5e], [D-v6e], [D-v5p], [D-7x], [G-8dd]. Book: [SB2]. The book's figures are about 10% below the docs-derived ones. This course uses the docs and notes where a book answer depends on the book's value. TPU 8t's ICI is "2× Ironwood", with no absolute figure published.

The number the collective formulas in Chapter 5 use is $W_\text{ici}$: the bandwidth one chip can push along one axis of the torus in a bidirectional ring. On an axis, a chip has one link in each direction, so $W_\text{ici}$ is two links times the one-way rate. On Ironwood that is 200 GB/s per axis derived. The Scaling Book uses 180 GB/s.

Topology by generation

v5e and v6e: a fixed 16×16 2D torus. Each chip has 4 links, to its neighbours in X and Y. A pod is 256 chips and cannot grow beyond that over ICI. Wraparound links exist only on an axis of the full length 16, so a 16×16 slice wraps both ways, an 8×16 slice wraps only on its 16-long axis, and a 4×4 slice does not wrap at all [SB2].

v4, v5p and Ironwood: cubes joined by optical switches. The physical unit is a 4×4×4 cube of 64 chips, one rack. Within a cube, chips connect over copper. Each face of a cube has 16 optical links, and the faces of all cubes in the pod run to a set of optical circuit switches [P-v4, §2.1] [D-arch]. Setting the switches joins cubes into any slice shape that is a multiple of 4 on each axis, with wraparound on every axis [SB2]. Ironwood supports slices from 2×2×1 up to 9,216 chips; slices larger than 2,048 chips (8×16×16) need Google account-team approval [D-7x].

Twisted tori. For some shapes, the switches can connect the wraparound links with an offset, like a Möbius strip, which shortens the average distance. The v4 docs report about 70% more bisection bandwidth for a twisted 4×4×8 and 40% for 4×8×8 [D-v4]. About 28% of large v4 slices used a twisted shape [P-v4, §2].

TPU 8i: Boardfly. Covered in its own section below.

Why wraparound matters

On a ring of $N$ chips with wraparound, the farthest chip is $N/2$ hops away and the average distance is $N/4$. Without wraparound the farthest is $N - 1$. More important for collectives: a ring algorithm needs the ring to close. Without the closing link, a bidirectional ring collective degrades, and the Scaling Book's rule of thumb is that a slice without wraparound "typically doubles the time of most communication" [SB2]. That is why a 2×2×2 Ironwood slice gets roughly half the collective bandwidth per chip of a full 4×4×4 cube.

Computed in scripts/ch03_ici.py.

Optical circuit switches

An optical circuit switch (OCS) redirects light from input fibres to output fibres with steerable MEMS mirrors. It does not read packets, so it adds no per-packet latency and has no buffers. It sets up circuits, and reconfiguring them takes milliseconds. The v4 switch, "Palomar", has 136 ports, 128 of them used. Circulators let one fibre carry both directions, halving the port count. A 4,096-chip v4 pod uses 48 switches [P-v4, §2.1–2.2].

What the switches buy:

• Slices of any shape. A slice is a set of cubes wired together by switch settings. The cubes do not need to sit next to each other on the floor.
• Failure isolation. A rack with a failed host or chip is switched out and replaced by a spare. Google's NSDI paper on v4 operations reports 99.98% system availability, with about 1% of jobs seeing a hardware outage [P-Resil]. The SIGCOMM paper reports up to 3× higher availability and up to 3.3× better model-dependent performance than a static fabric [P-Lightwave].
• Spares in the pod. Ironwood pods have 9,216 chips; the Hot Chips coverage attributes the choice of 9,216 over 8,192 to spare racks [HC-7x].

What they cost: under 5% of system cost and under 3% of power, per the v4 paper. The SIGCOMM paper, counting differently, says under 6% [P-v4] [P-Lightwave].

Hosts, PCIe and DCN

Chips sit four to a tray, and a host CPU attaches to its tray over PCIe [SB2]. v5e and v6e hosts drive 8 chips; v4, v5p and Ironwood hosts drive 4 [D-v5e] [D-7x]. A 4-chip Ironwood VM has 224 vCPUs and 960 GB of RAM [D-7x].

Slices connect to each other through the hosts over the datacenter network, DCN: chip, PCIe, host NIC, network, remote NIC, PCIe, chip [SB2]. This is multislice. DCN bandwidth is 50 Gb/s per chip on v5p and 100 Gb/s on v6e and Ironwood, that is 6.25 and 12.5 GB/s [D-v5p] [D-7x]. TPU 8t's new Virgo network raises DCN up to 4× Ironwood and connects more than 134,000 chips in one fabric, with 47 Pb/s of bisection bandwidth [G-8dd].

The ladder sets where each kind of parallelism belongs. Tensor parallelism exchanges activations every layer and needs ICI. FSDP gathers weights every layer and needs ICI. Pure data parallelism exchanges gradients once per step, can overlap them with the backward pass, and tolerates DCN. MaxText encodes exactly this split: every parallelism axis exists twice, as ici_*_parallelism within a slice and dcn_*_parallelism across slices. Its default config comments dcn_tensor_parallelism as "never recommended" [R-maxtext]. Chapter 6 derives the batch size needed for DCN data parallelism.

Pathways. Classic multi-host JAX runs the same Python program on every host, a multi-controller model. Pathways instead runs one controller that dispatches compiled programs asynchronously to groups of chips connected by ICI ("islands"), which it calls across DCN [P-Pathways, §2, §4]. It reported close to 100% accelerator utilization on SPMD workloads at 2,048 TPUs. Google's 8t announcement describes scaling to more than a million chips per cluster with JAX and Pathways [G-8dd].

Latency, bandwidth, and why 8i left the torus

Every ICI hop costs about 1 µs, whatever the message size [SB2]. A transfer's time is the sum of a latency term and a bandwidth term:

Each hop adds a fixed microsecond; the bytes then flow at the bandwidth of the links carrying them.

Now shrink the message. A decode step with a small batch moves kilobytes per collective, not megabytes. On v5e, below about 45 kB per hop, the latency term exceeds the bandwidth term [SB3]. In that regime, time is hop count times 1 µs, and the torus's diameter is the problem: 16 hops across an 8×8×16 slice.

TPU 8i's answer is Boardfly, a hierarchical topology in the dragonfly family [G-8dd]:

• 4 chips on a board form the basic unit.
• 8 boards form a group, fully connected to each other.
• 36 groups connect through optical circuit switches, for 1,152 chips. Google says up to 1,024 are active, and the rest are presumably spares.
• The largest chip-to-chip distance is 7 hops, against 16 for a 1,024-chip 8×8×16 torus. Google describes this as cutting the network diameter by more than 50%.

Combined with the Collectives Acceleration Engine (Chapter 2), the design targets the latency-bound AllReduce and AllToAll traffic of low-batch MoE decoding, rather than the bandwidth-bound gradient traffic of training. 8t, the training chip, keeps the 3D torus.

Pod accounting

Adapted from the Scaling Book's Chapter 2, problem 2 [SB2], with Ironwood added. Computed in scripts/ch03_ici.py.

Ironwood's FP8 peak for the pod is $4.25 \times 10^{19}$, matching Google's 42.5 exaFLOPS [G-7x]. Google's announcement gives 1.77 PB of HBM per pod, which is 9,216 × 192 GB counted in decimal units [G-7x-ga]. The docs give 192 GiB per chip, which would make it 1.90 PB [D-7x]. Unit labels in Google's materials are inconsistent, and the true figure is unverified.