Inference rooflines

Tick each objective you can deliver out loud without notes. Then write the one-paragraph explanation you would give a room of GPU engineers. Saved attempts stay below, so you can compare them over the week.

• I can state the decode step-time lower bound and the general step-time formula, and compute both for a model on a slice.
• I can compute the compute-bound decode batch for each weight and math precision on v5e, Ironwood, 8i and H100, and explain why KV growth makes it unreachable at long context.
• I can explain why prefill is compute-bound and decode is not, and what that implies for disaggregated serving and the prefill-to-decode server ratio.
• I can choose a decode sharding — model parallelism up to F/(Bβ), the KV cache split by heads then by batch, never FSDP — and say when its collectives are latency-bound.
• I can explain int8, fp8 and fp4 weights, activations and KV on TPU, say which generations have native fp8, and describe how Qwix applies post-training quantization.
• I can explain speculative decoding as spending idle FLOPs in a memory-bound step.

Prompts to answer out loud

1. Your PM wants 2× decode throughput on the same chips. List what moves the number and by how much — batch, fp8 weights, fp8 KV, more TP, speculative decoding.
2. Why is FSDP an order-of-magnitude regression for decode, when it is free for training?
3. At 32k context, why does 8i's FP8 peak barely matter for decode, and what does matter?
4. How many prefill servers do you need per decode server for 8k prompts and 512-token answers, and why?