Choosing a mesh for training

Tick each objective you can deliver out loud without notes. Then write the one-paragraph explanation you would give a room of GPU engineers. Saved attempts stay below, so you can compare them over the week.

• I can derive the condition under which each training scheme is compute-bound — data parallelism, FSDP, tensor parallelism, FSDP plus TP, and cross-slice data parallelism over DCN.
• I can compute the optimal FSDP degree, X_opt, and pick a mesh for a given model, batch and slice.
• I can redo the Scaling Book's LLaMA-3-70B plan on a v5p pod, and explain why the same batch is communication-bound on a full Ironwood pod.
• I can explain why pipeline parallelism is central to GPU recipes and secondary on TPUs, citing LLaMA-3's actual GPU configuration.
• I can map these choices to MaxText configuration — ici_* against dcn_* axes and logical axis rules.

Prompts to answer out loud

1. Give the one-line rule for when FSDP stops being free, with the number for v5p and for Ironwood.
2. Why did Meta use 16-way pipeline parallelism for LLaMA-3 405B on GPUs, when a TPU pod would mostly not?
3. Choose a mesh for a 70B dense model on a 2,048-chip Ironwood slice with a 4M-token batch, and say whether it is compute-bound.
4. What would you change if your Ironwood training run is communication-bound and you cannot raise the batch?