TPU Deep Dive
Course
Review
Slides
Exams
Sources
Theme: system
Day 3 exam
Sharding and training 12 questions drawn.
Draw a new set
Question 1 of 12
Recall
In a TPU profile, an all-gather-start op is followed by a dot fusion and then all-gather-done. What does that show?
1
The AllGather is asynchronous and overlaps with the matmul scheduled between start and done.
2
The AllGather ran twice.
3
The matmul is waiting for the AllGather.
4
The collective was offloaded to the host.
Check answer
Enter
Press 1–4 to choose.