Model Distillation: How Small Models Learn to Punch Above Their Weight

Model distillation is one of the most practically important techniques in modern AI, and one of the most chronically under-explained. Most coverage treats it as a footnote to compression research or blurs it together with quantization and pruning. None of those are the same thing.

Soft labels carry more signal#

Training a network from scratch means minimizing the error between its predictions and hard labels: a cat photo gets a 1 for cat and 0 for everything else. Distillation, formalized in a 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean (building on earlier model-compression work by Bucilua and Caruana), replaces that hard supervision with the output distribution of a larger, already-trained model, the teacher. The smaller model being trained is the student.

This matters because teacher outputs are soft labels. Shown a cat, a large model does not output 100% cat. It might output cat 82%, lynx 11%, dog 4%, fox 2%, and a long tail. That distribution encodes the teacher's sense of which classes are similar and how confident it is, structure a raw one-hot label throws away. Hinton called it dark knowledge: the information hidden not in the top answer but in the ratios of all the answers the teacher did not pick. A student trained on soft labels learns that cats sit closer to lynxes than to dogs, and that this particular image blurs the boundary.

Temperature scaling: the mechanism behind the signal#

There is a catch: a well-trained teacher is often highly confident anyway, maybe 99.7% cat, which is barely softer than a hard label. Distillation fixes this with temperature scaling. The teacher's logits (its raw pre-softmax scores) are divided by a temperature T before the softmax. At T=1 you get the normal distribution; at T=4 or T=8 it flattens, amplifying the minority probabilities so the buried structure becomes legible. Both teacher and student use the same temperature during distillation, and the student's loss is a weighted sum of the ordinary cross-entropy against the true labels and the KL divergence between the two softened distributions. One practical detail: softening by T shrinks the distillation gradients by about T squared, so implementations multiply that term by T squared to keep it balanced. Temperature is reset to 1 at inference; it is purely a training-time lever.

Why distilled beats scratch-trained at the same size#

If you could just train a small model from scratch on the same data, why is a distilled version of the same architecture better? Training-signal density. A hard-labeled example gives the student one correct answer and nothing else; the teacher turns that same example into a full probability vector, the answer plus its entire uncertainty landscape over every other class. The student learns from far more information per gradient step without any more labeled data. There is also a regularization effect: soft labels discourage the student from slamming similar-class outputs to zero, which smooths decision boundaries and improves out-of-distribution behavior. The clearest recent demonstration is DeepSeek's R1 work: distilling the large model's reasoning traces into small dense students (Qwen and Llama backbones, roughly 1.5B to 70B) produced a stronger small reasoner than running large-scale reinforcement learning directly on that same small model.

Where it pays off#

Distillation's value shows up under scale or hardware constraints. On-device inference: a teacher runs on datacenter GPUs while its distilled student runs on the NPU in a phone or laptop, landing far closer to the teacher than a naively-trained small model would, which is what makes offline features viable at all. Latency-critical paths: a distilled student a fraction of the size answers several times faster, sometimes an order of magnitude, at a modest quality cost, which is the right trade for autocomplete, routing, classification, and moderation. Cost: a model that reaches most of the teacher's quality at a small fraction of the compute is, for most production workloads, the optimal operating point, with the teacher reserved for the queries that genuinely need it. And a general model distilled over your own domain data is often smaller, faster, and better than the general teacher on your specific task.

Distillation in the LLM era#

The technique shifts shape at LLM scale. Sequence-level distillation trains the student on the teacher's actual generations rather than its full next-token distribution, sidestepping the need for teacher logits over a 100,000-plus-token vocabulary at every step, at the cost of some richness. On-policy distillation goes further: the student generates, then the teacher scores or corrects the student's own sequences, so the model learns to recover from its own mistakes instead of only ever seeing clean teacher text. Synthetic data and self-play blur the line between distillation and synthetic-data training, but the knowledge-transfer mechanism is the same; Google has said its open Gemma models are trained using distillation from larger Gemini-class teachers, and the DeepSeek-R1-Distill line productizes the pattern. When you can only see a commercial model's outputs and not its logits, you are limited to behavioral cloning, which still beats human labels alone but comes with two caveats worth stating plainly: most providers' terms of service prohibit using their outputs to train competing models, and the student inherits the teacher's blind spots with none of the transparency of an auditable training set.

Trade-offs and failure modes#

Distillation is not free. Teacher quality is a hard ceiling: a miscalibrated or biased teacher produces bad soft labels the student inherits, often without the auditability of hand-labeled data. Capacity mismatch: if the student is far too small for the task, distillation improves the exchange rate but cannot repeal the physics of parameter count. Distribution shift: soft labels are only informative where the teacher actually understands the data, so if your deployment distribution drifts, the teacher's confidence can mislead as easily as guide. And there is training-time overhead: you must run the teacher over your training data to produce the labels or generations, a one-time cost that is usually amortized many times over across the student's inference lifetime.