TPU Deep Dive
Course
Review
Slides
Exams
Sources
Theme: system
Sources
SB2
How to Think About TPUs
What is a TPU
SB2
How to Think About TPUs
Appendix A: VPU, scalar core
SB2
How to Think About TPUs
Appendix B: systolic arrays
SB2
How to Think About TPUs
Worked problems
P-Norrie
Norrie et al., "The Design Process for Google's Training Chips: TPUv2 and TPUv3"
scalar, vector and matrix units
P-v4
Jouppi et al., "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings"
§2.2 chip, §3 SparseCore
D-arch
System architecture
D-7x
TPU7x (Ironwood)
J-hw
Pallas "TPU Hardware Reference"
X-tiled
Tiled layout
PL-tpu
Writing TPU kernels
PL-sc
SparseCore kernels
G-8dd
Gupta & Mugazambi, "TPU 8t and TPU 8i technical deep dive"
PL-pipe
TPU pipelining
megacore dimension semantics
T-exec
TorchTPU docs: Execution model
compile cost
SB12
How to Think About GPUs
GPUs vs TPUs at the chip level
N-tune
Hopper tuning guide
§1.4.1, occupancy and SMEM
A-trn2-nki
Trainium2 architecture guide for NKI
Chapter 2. Day 1, second block, 55 minutes
Anatomy of a TensorCore
Read
Quiz
12
Flashcards
20
Teach-back
8
20
left in this session,
20
of them new.
Reviewed
0
Question
A TPU TensorCore is the counterpart of which GPU unit, and how many are there per chip?
Show answer
Space