[LLM 7/10] Model Distillation: The Teacher's Wrong Answers Are the Most Valuable Part

In chapter 6 we distilled a context into the weights of the same model. In this chapter we distill an entire model into a smaller one. The two chapters are a deliberate pair: context distillation changes what the model knows without having to tell it anymore — model distillation changes the size of the model while trying not to change what it can do. And the heart of this chapter cuts hard against intuition: the most valuable thing a teacher can hand its student is not the right answer, but the way the teacher is wrong — and the dial called temperature is what makes that visible.

Open in Colab07_model_distillation.ipynb

Thai LLM seriesPart 7 of 10

1. 1Continue Pretraining
2. 2SFT and LoRA
3. 3RLHF and PPO
4. 4DPO: Direct Preference Optimization
5. 5GRPO
6. 6Context Distillation
7. 7Model Distillation(you are here)
8. 8Guardrails
9. 9Benchmarking
10. 10Deployment

1. The Problem

Suppose you've walked the full six chapters and arrived at a Qwen3-1.7B that handles your organization's Thai workload satisfyingly well. Then one day the infrastructure team asks the one question the model can't answer: "What's the GPU bill per month?"

The 1.7B model eats nearly 3× the VRAM of the 0.6B and answers about 2.5× slower. At 100 requests per second, that difference is not a detail — it's the number of extra cards you buy every month for the life of the system.

Option	Quality	Cost at serving time
Deploy the 1.7B teacher directly	Best	~3× VRAM, ~2.5× slower, paid for the system's whole life
Deploy the 0.6B student directly	Clearly worse	Cheap and fast
SFT the student on gold answers	Somewhat better	Cheap and fast
Model distillation	Moves toward the teacher	Cheap and fast, exactly like the student

The question is what the last row knows that the SFT row doesn't — same training, same data, so where's the difference?

The answer is in the amount of information per token. A hard label is a one-hot vector: it says "the answer is 3," full stop. But the teacher's distribution, softened by temperature, says "the answer is 3 — but 8 is somewhat plausible too, and 'cat' is complete nonsense." This ranking over all the wrong answers is what Hinton called dark knowledge. It encodes the similarity structure of the world (the digit 3 is closer to 8 than to a cat) and it vanishes entirely the moment you keep only the argmax.

At every token position the teacher has 151,936 numbers to offer (Qwen3's vocab size) — a hard label keeps exactly one.

2. What We're Going to Do

We take Qwen/Qwen3-1.7B as the teacher and Qwen/Qwen3-0.6B-Base as the student, then distill at three levels that pass increasingly fine-grained information from the teacher:

1. SeqKD — have the teacher generate answers, then SFT the student on them (a sample from the distribution)
2. Logit KD — have the student imitate the teacher's entire distribution, token position by token position (the distribution itself)
3. GKD (optional extra) — have the student sample its own answers and let the teacher score the distributions on-policy

And the piece we cannot do without is the control row: SFT the student on the gold answers, same data, same number of steps. Without that row, we have no way to tell whether the gains came from "the teacher's distribution" or from "just training more."

The core idea of this chapter

A hard label says "the answer is 3" — the teacher's distribution says "3, but 8 is almost right, and 'cat' is impossible." The ranking over wrong answers is dark knowledge, and temperature is the dial that reveals it. At T = 1 this knowledge is squeezed out of sight (the teacher is 0.97 confident); raise T and it becomes a signal you can actually train on.

This is not a laboratory curiosity — it's how most of the world's "small but capable" models are actually built. The Qwen3-0.6B we've used all series long was itself trained with strong-to-weak distillation from the larger members of its own family. In this chapter we're doing the same thing, at a scale free Colab can handle.

3. The Equations

3.1 Hinton's KD loss

$$\mathcal{L}_{\text{KD}} = (1-\alpha)\,\mathcal{L}_{\text{CE}}\big(y,\ \text{softmax}(z_S)\big) + \alpha\, T^2\,\mathbb{D}_{\text{KL}}\Big(\text{softmax}(z_T/T)\ \big\|\ \text{softmax}(z_S/T)\Big)$$

• $z_S, z_T$ = the student's and teacher's logits, at the same token position, on the same input
• $y$ = the gold answer (hard label) — the first term is ordinary cross-entropy, identical to SFT
• $T$ = temperature, divided into the logits on both sides before the softmax
• $\alpha$ = the weight of the soft term (we use 0.9 — listen mostly to the teacher, with the gold answer as a safety line)

Notice the direction of the KL: the teacher comes first. This is forward KL, which forces the student to spread its probability over everywhere the teacher puts weight. Hold on to that — equation 3.4 is about to turn it into "one point on a line."

Now, where does the $T^2$ sitting in front of the KL come from? Nearly every piece of KD code on the internet has this factor, but very few explain why — and if you don't understand it, you will tune T wrongly without ever knowing.

3.2 Deriving where $T^2$ comes from — the two lines that separate "understood" from "copied"

Line 1 — the gradient of the soft term with respect to one student logit is the standard softmax-CE gradient plus the chain rule through $z/T$, which emits one factor of $1/T$:

$$\frac{\partial \mathcal{L}_{\text{soft}}}{\partial z_{S,i}} = \frac{1}{T}\Big(q_i^{(T)} - p_i^{(T)}\Big), \qquad q^{(T)} = \text{softmax}(z_S/T),\quad p^{(T)} = \text{softmax}(z_T/T)$$

Line 2 — as $T$ grows, the softmax flattens toward uniform: $\text{softmax}(z/T)_i \approx \tfrac{1}{K} + \tfrac{z_i - \bar z}{KT}$ (with $K$ = vocab size). So the difference $q^{(T)}_i - p^{(T)}_i$ shrinks by another factor of $1/T$:

$$\frac{\partial \mathcal{L}_{\text{soft}}}{\partial z_{S,i}} \approx \frac{(z_{S,i} - \bar z_S) - (z_{T,i} - \bar z_T)}{K\,T^2} \;\propto\; \frac{1}{T^2}$$

The soft term's gradient scales as $1/T^2$, while the hard CE term doesn't depend on $T$ at all. Without multiplying the $T^2$ back in, moving T from 1 to 4 secretly divides the soft term's learning rate by ~16. You'd conclude "high T doesn't work," when in reality you just quietly switched off your own soft loss. Multiplying by $T^2$ makes the gradient scale nearly independent of T — so the $\alpha$ you tuned keeps the same meaning at every T.

A bonus from line 2: in the same limit, the soft loss reduces to matching mean-centered logits in an MSE-like sense — KD is "soft logit regression" that weights the head of the distribution more than the tail.

3.3 SeqKD — the cheap baseline (Kim & Rush, 2016)

$$\mathcal{L}_{\text{SeqKD}} = -\,\mathbb{E}_{\hat y \sim \pi_T}\left[\sum_t \log \pi_S(\hat y_t \mid x, \hat y_{<t})\right]$$

Look familiar? This is plain SFT on answers the teacher generated — nothing more. Instead of sending the whole distribution, the teacher sends "one sample" drawn from its own distribution.

An advantage that often goes unnoticed: SeqKD doesn't care whether the tokenizers match, because it sends text, not logits. Many open-source models advertised as "distilled from GPT-4" are in fact pure SeqKD — collect the teacher's answers through an API, then SFT. That's why it's the baseline we must measure before reaching for anything more expensive.

3.4 GKD and generalized JSD — the line connecting this chapter to chapter 6

Logit KD per equation 3.1 has a structural weakness: the student learns on sentences someone else wrote (teacher forcing), but at serving time it must generate continuations of its own answers — the accumulating mismatch is called exposure bias. GKD (Agarwal et al., 2023) fixes this by letting the student sample its own answers and having the teacher score those tokens, while also generalizing the distance between distributions to:

$$\mathbb{D}^{(\beta)}_{\text{JSD}}(P\,\|\,Q) = \beta\,\mathbb{D}_{\text{KL}}(P\,\|\,M) + (1-\beta)\,\mathbb{D}_{\text{KL}}(Q\,\|\,M), \qquad M = \beta P + (1-\beta) Q$$

with $P$ = teacher, $Q$ = student. The value $\beta$ sweeps from one pole to the other:

• $\beta \to 0$ → forward KL $\mathbb{D}_{\text{KL}}(P\|Q)$ — mass-covering: the student must spread over every mode of the teacher
• $\beta \to 1$ → reverse KL $\mathbb{D}_{\text{KL}}(Q\|P)$ — mode-seeking: the student commits to the modes it can actually handle

To say it as plainly as possible: the reverse KL that chapter 6 chose isn't an oddity from another world — it's the point $\beta = 1$ on this very line, and Hinton's classic KD is the point $\beta = 0$. The two chapters are members of the same family, differing only in "who has to move toward whom" — a student much smaller than its teacher usually benefits from the mode-seeking side, because it never had the capacity to cover every one of the teacher's modes anyway.

The full lesson is in the course

This post is roughly the first 30% of the chapter. The rest — environment setup, data preparation, the main code, measured results and the wrap-up — is in the free LLM Finetuning course. Sign in with Google to read it.

Read the full lesson in the course →