Anatomy of a TensorCore

Tick each objective you can deliver out loud without notes. Then write the one-paragraph explanation you would give a room of GPU engineers. Saved attempts stay below, so you can compare them over the week.

• I can draw a TensorCore and say what each unit does — scalar unit, VPU, MXU, XLU, VMEM, SMEM, vector registers and DMA engines — and which one issues instructions to the rest.
• I can explain weight-stationary systolic dataflow and compute an MXU tile's utilization from M, K, N and the array size.
• I can explain why a contraction or output dimension smaller than the array, or not a multiple of it, wastes MXU capacity, and estimate the waste.
• I can explain why VMEM is a compiler-managed scratchpad rather than a cache, give its size per generation, and compute the ridge point for VMEM-resident weights.
• I can explain (8,128) tiling and the bf16 and int8 packing, and what they imply for array shapes.
• I can say how many devices JAX sees per chip on v4, v5e, v5p, v6e, Ironwood, 8t and 8i, and why.
• I can say what SparseCores and 8i's Collectives Acceleration Engine are for.
• I can map SM, warp scheduler, CUDA core, shared memory, Tensor Core and HBM onto TPU units, and name three places the mapping breaks.

Prompts to answer out loud

1. A GPU hides memory latency with up to 64 resident warps per SM. What hides it on a TPU, and what does that imply for who has to be clever?
2. Google moved from 128×128 to 256×256 matrix units in v6e. What does that buy, and what does it cost a model with head dimension 64?
3. What is VMEM, how big is it on Ironwood and on 8i, and why did 8i triple it?
4. Why does a decode step leave the matrix units mostly idle even if the weights were already on chip?