TPU Deep Dive
Course
Review
Slides
Exams
Sources
Theme: system
Sources
SB12
How to Think About GPUs
Chip level
SB12
How to Think About GPUs
Networking
SB12
How to Think About GPUs
Collectives
SB12
How to Think About GPUs
LLM rooflines
SB12
How to Think About GPUs
GB200
SB2
How to Think About TPUs
TPU networking
SB3
Sharded Matrices and How to Multiply Them
TPU collective reach
N-wp
H100 architecture whitepaper
H100 in depth
N-tune
Hopper tuning guide
§1.4
N-gb200
GB200 NVL72
N-rubin
Vera Rubin NVL72
A-trn2
Trainium2 chip architecture
A-trn2-nki
Trainium2 architecture guide for NKI
A-trn2-sys
Trn2 instance and UltraServer architecture
A-trn3
Trainium3 chip architecture
A-trn3-sys
Trn3 UltraServer architecture
A-trn4
Trainium4 + NVLink Fusion preview
A-torch
PyTorch on Neuron / TorchNeuron
G-8dd
Gupta & Mugazambi, "TPU 8t and TPU 8i technical deep dive"
J-hw
Pallas "TPU Hardware Reference"
T-pt
PyTorch blog, "Introducing TorchTPU"
A TPU primer
N-graphs
CUDA Programming Guide §4.2, CUDA Graphs
SB8
Serving LLaMA 3-70B on TPUs
FLOPs per dollar
Chapter 11. Synthesis, after Day 5, 55 minutes
TPU and GPU architecture, side by side
Read
Quiz
12
Flashcards
20
Teach-back
6
Question 1 of 12
Recall
Which pairing of design bet and cost is correct?
1
TPU: few single-threaded cores, a compiler-managed scratchpad and whole-program compilation; cost: static shapes and poor tolerance for irregular access.
2
GPU: few large cores; cost: low flexibility.
3
TPU: hardware caches; cost: unpredictable performance.
4
Trainium: one core per chip; cost: no matrix unit.
Check answer
Enter
Press 1–4 to choose.