TPU Deep Dive
Course
Review
Slides
Exams
Sources
Theme: system
Sources
SB2
How to Think About TPUs
TPU networking
SB2
How to Think About TPUs
TPU specs, ICI table
SB2
How to Think About TPUs
Worked problems 2, 5, 6
D-arch
System architecture
slices, cubes, OCS, twisted torus
D-v4
TPU v4
twisted torus bisection
D-v5e
TPU v5e
D-v5p
TPU v5p
D-v6e
TPU v6e
D-7x
TPU7x (Ironwood)
P-v4
Jouppi et al., "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings"
§2 OCS and availability
P-Lightwave
Liu et al., "Lightwave Fabrics: At-Scale Optical Circuit Switching for Datacenter and ML Systems"
P-Resil
Zu et al., "Resiliency at Scale: Managing Google's TPUv4 Machine Learning Supercomputer"
G-7x
Ironwood: the first Google TPU for the age of inference
G-7x-ga
Ironwood TPUs and new Axion-based VMs
HC-7x
Ironwood at Hot Chips 2025, as reported by ServeTheHome
pod size and spares
SB3
Sharded Matrices and How to Multiply Them
latency-bound transfers
G-8dd
Gupta & Mugazambi, "TPU 8t and TPU 8i technical deep dive"
Boardfly vs torus math
P-Pathways
Barham et al., "Pathways: Asynchronous Distributed Dataflow for ML"
§2, §4
R-maxtext
MaxText repo
ici_* and dcn_* parallelism
SB12
How to Think About GPUs
GPU networking
N-gb200
GB200 NVL72
A-trn2-sys
Trn2 instance and UltraServer architecture
A-trn3-sys
Trn3 UltraServer architecture
Chapter 3. Day 2, first block, 55 minutes
ICI, OCS, slices and pods
Read
Quiz
12
Flashcards
21
Teach-back
7
21
left in this session,
21
of them new.
Reviewed
0
Question
What is ICI?
Show answer
Space