TPU Deep Dive
Course
Review
Slides
Exams
Sources
Theme: system
Sources
P-v1
Jouppi et al., "In-Datacenter Performance Analysis of a Tensor Processing Unit"
§2 architecture, §4 rooflines
P-v23
Jouppi et al., "A Domain-Specific Supercomputer for Training Deep Neural Networks"
Table 3, v1 to v3
P-Norrie
Norrie et al., "The Design Process for Google's Training Chips: TPUv2 and TPUv3"
Derivation from TPUv1
HC-v23
Norrie & Patil, "Google's Training Chips Revealed: TPUv2 and TPUv3"
Why 128×128
P-v4
Jouppi et al., "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings"
§2 OCS, §3 SparseCore
D-v5e
TPU v5e
D-v5p
TPU v5p
D-v6e
TPU v6e
G-v6e
Introducing Trillium, 6th-gen TPUs
D-7x
TPU7x (Ironwood)
G-7x
Ironwood: the first Google TPU for the age of inference
G-8dd
Gupta & Mugazambi, "TPU 8t and TPU 8i technical deep dive"
J-hw
Pallas "TPU Hardware Reference"
per-core table, TPU 8 BF16/FP8
P-Micro26
Jouppi, Patterson, Lakshmanamurthy, Young, "Google's Training Supercomputers from TPU v2 to Ironwood…"
abstract
SB1
All About Rooflines
Where does the time go
SB1
All About Rooflines
Matrix multiplication
SB1
All About Rooflines
Network rooflines
SB2
How to Think About TPUs
TPU specs
SB6
Training LLaMA 3 on TPUs
LLaMA 3 shapes
N-h100
H100 product page
Chapter 1. Day 1, first block, 55 minutes
TPU lineage and roofline thinking
Read
Quiz
12
Flashcards
18
Teach-back
6
Question 1 of 12
Recall
A chip's ridge point against HBM is:
1
Peak FLOP/s divided by HBM bandwidth in bytes/s.
2
HBM capacity divided by peak FLOP/s.
3
The batch size at which HBM fills up.
4
Peak FLOP/s divided by ICI bandwidth.
Check answer
Enter
Press 1–4 to choose.