Post-training & Efficiency

Post-training & Efficiency

Why it matters

A pretrained language model predicts text. Post-training turns it into an assistant that follows instructions, reasons and reflects human preferences, and efficiency techniques make it affordable to fine-tune and serve. Labs ask about both: how GRPO and DPO differ from PPO-RLHF, and how quantization trades accuracy for memory.

Core ideas: preference and reward optimization

Method	Idea	Objective (maximize) or loss (minimize)	Practiced in
Reward model	Score responses, trained on human preference pairs (Bradley–Terry)	loss -log σ(r_chosen - r_rejected)	Reinforcement Learning
PPO-RLHF	Maximize reward with PPO; a KL term keeps π_θ near the reference model	clipped surrogate with A = r - β·KL	Reinforcement Learning
DPO	No reward model and no sampling: optimize preferences through log-ratios	loss -log σ(β·(Δ_chosen - Δ_rejected))	Reinforcement Learning
GRPO	PPO without a value network: the group of samples for a prompt is the baseline	clipped surrogate with A = (r - mean) / std, minus β·KL_k3	this section

Why a KL term? Without it the policy drifts toward outputs that fool the reward model ("reward hacking") and forgets how to write fluent text. The KL is estimated from the sampled tokens' log-probs, with kl_t = log π_θ(y_t) - log π_ref(y_t):

Estimator	Per token	Unbiased	Always ≥ 0	Used by
k1	kl_t	yes	no	InstructGPT-style KL-shaped reward
k3	exp(-kl_t) + kl_t - 1	yes	yes	GRPO

GRPO and verifiable rewards. GRPO (DeepSeekMath, 2024) is popular for reasoning models trained on verifiable rewards: an answer checker or unit tests give 1 for correct and 0 for wrong. For each prompt, sample G responses. The group mean replaces the critic, and dividing by the group std puts the advantages on a common scale. A prompt where every sample gets the same reward has zero advantage everywhere, so it contributes no gradient.

Core ideas: efficiency

Training and serving large models is dominated by memory. LoRA trains a small low-rank update instead of the full weights, and quantization stores weights in fewer bits.

Technique	Idea	Saves
LoRA	W + (α/r)·A·B with rank r ≪ d; only A and B are trained	optimizer state and gradients for the frozen W
Int8 absmax quantization	scale = max abs(w) / 127, store round(w / scale) as int8, one scale per output row	4× weight memory vs float32, fast int8 matmuls

One scale per output channel (row) limits the damage from outliers. A single huge weight only coarsens its own row.

Worked micro-example

GRPO advantages. Four samples for one prompt, verifier rewards [1, 0, 0, 1]: mean 0.5, population std 0.5, so the advantages are [+1, -1, -1, +1]. With rewards [1, 0, 0, 0] (std 0.433), the one correct answer gets +1.732 and each wrong one -0.577. A rare success earns a larger push.

Int8 absmax. Row [0.7, -1.2, 0.3]:

scale = 1.2 / 127 = 0.009449
w_q   = round([0.7, -1.2, 0.3] / scale) = [74, -127, 32]
w_hat = w_q · scale = [0.6992, -1.2, 0.3024]      max error 0.0024 ≤ scale / 2

Quantize [[0.01, -0.02, 0.013], [5.0, -3.0, 1.7]] with a single scale (5 / 127) and the first row becomes [0, -1, 0]: it is almost erased. Per-row scales keep it.

Common pitfalls

• GRPO std: use the population std (ddof=0, PyTorch unbiased=False) and add eps, so identical rewards give 0 instead of NaN.
• Masking: average token terms over real tokens only. Padding must not dilute a response's mean.
• KL counted twice: in PPO-RLHF the KL goes either into the reward or into the loss, never both.
• Sign conventions: objectives J are maximized; report and minimize loss = -J.
• Quantization: clip to [-127, 127] and guard all-zero rows (max(absmax, 1e-8)). Accumulate int8 products in int32, or they overflow.

Interview angle

• Why does GRPO not need a critic, and what does it cost? (G samples per prompt; variance from small groups.)
• What does the per-response 1/|o_i| normalization do to long vs short answers? (Dr. GRPO argues it biases length.)
• Compare DPO, which is offline, with PPO and GRPO, which are online and sample from the current policy.
• When does per-channel quantization beat per-tensor, and why do activation outliers (LLM.int8()) need special handling?
• Complexity: an int8 matmul costs the same O(n·d_in·d_out) as float, but moves 4× fewer weight bytes. On memory-bound decoding, that is where the speedup comes from.

Problems in this section

1. GRPO: Group-Relative Advantages & Objective: group-normalized advantages, then the clipped, masked, k3-penalized GRPO objective.
2. Int8 Weight Quantization (absmax): per-channel and per-tensor quantization, dequantization and an int8 matmul with int32 accumulation.

The reward-model loss, the KL penalty, the PPO-RLHF objective and DPO are practiced in Reinforcement Learning (start with PPO Clipped Objective and KL Divergence Penalty (RLHF), which GRPO builds on). The LoRA adapter is in PyTorch Basics.