LLM inference and training internals

The techniques behind production language models, one at a time: sampling strategies, the KV cache, grouped-query attention, the SwiGLU block, the FlashAttention softmax trick, mixture-of-experts routing, speculative decoding, preference optimisation and quantisation.

12 problems · about 5.5 h · Advanced. You'll need: The Build a transformer path, or equivalent experience.

By the end you can

• Implement the decoding strategies and explain how each trades quality for diversity
• Explain where inference time and memory go, and how the KV cache, GQA and speculative decoding reduce them
• Write the DPO and GRPO objectives used to align models after pre-training

Steps

1. Step 1: Temperature Scaling → Temperature: reshape the next-token distribution.
2. Step 2: Nucleus (Top-p) Sampling → Nucleus sampling and its edge cases.
3. Step 3: KV Cache for Efficient Generation → Cache keys and values so each new token costs O(n), not O(n²).
4. Step 4: Grouped Query Attention (GQA) → Share key/value heads to shrink the KV cache.
5. Step 5: SwiGLU Feed-Forward Block (LLaMA MLP) → The SwiGLU feed-forward block used in LLaMA-style models.
6. Step 6: Online Softmax (the FlashAttention Trick) → The one-pass softmax that makes FlashAttention possible.
7. Step 7: Mixture-of-Experts Routing → Top-k expert routing and load balancing in mixture-of-experts.
8. Step 8: Speculative Decoding → Draft with a small model, verify with the large one.
9. Step 9: Token Cross-Entropy and Perplexity → Token cross-entropy and perplexity, the standard LM metric.
10. Step 10: Direct Preference Optimization (DPO) → Direct preference optimisation: align without a reward model.
11. Step 11: GRPO: Group-Relative Advantages & Objective → GRPO's group-relative advantages, used for reasoning models.
12. Step 12: Int8 Weight Quantization (absmax) → Absmax int8 quantisation and the error it introduces.