TPU Deep Dive
Course
Review
Slides
Exams
Sources
Theme: system
Sources
SB7
All About Transformer Inference
Basics
SB7
All About Transformer Inference
Linear ops
SB7
All About Transformer Inference
Latency and throughput bounds
SB7
All About Transformer Inference
LLaMA-2-13B tables
SB7
All About Transformer Inference
Tricks
SB7
All About Transformer Inference
Sharding inference
SB7
All About Transformer Inference
Engine design
SB7
All About Transformer Inference
Latency-bound comms
SB7
All About Transformer Inference
Speculative sampling
SB7
All About Transformer Inference
Problems
SB8
Serving LLaMA 3-70B on TPUs
Serving problems
P-ESTI
Pope et al., "Efficiently Scaling Transformer Inference"
R-qwix
Qwix
R-tpuinf
tpu-inference (being renamed vLLM JAX)
Qwix config in tpu-inference
D-arch
System architecture
dtypes per generation
D-v5p
TPU v5p
D-7x
TPU7x (Ironwood)
G-8dd
Gupta & Mugazambi, "TPU 8t and TPU 8i technical deep dive"
J-hw
Pallas "TPU Hardware Reference"
Chapter 7. Day 4, first block, 55 minutes
Inference rooflines
Read
Quiz
12
Flashcards
20
Teach-back
6
20
left in this session,
20
of them new.
Reviewed
0
Question
Prefill against decode, in one line each?
Show answer
Space