Probability assigned to the correct class.
One example, three probability distributions
Teacher soft targetHard labelStudent output
L = 0.35 · CE + 0.65 · T² · KLMultiplying the soft loss by T² keeps its gradient scale comparable as temperature changes.
Remaining mismatch in softened distributions.
Similarity among the teacher's non-winning classes.
Illustrative parameter ratio from capacity.
Temperature reveals relationships
Dividing logits by a temperature above one flattens the teacher distribution. The top class remains important, while secondary classes become visible enough to carry similarity information into the student.
Two losses serve different jobs
Cross-entropy anchors the true label. KL divergence transfers the teacher's relative preferences. The mixing weight decides whether the student follows ground truth, teacher judgment, or a negotiated combination.
Capacity sets a ceiling
A smaller model cannot reproduce every teacher distinction. Distillation can improve its use of limited capacity, but it does not make architecture, data coverage, latency, or evaluation tradeoffs disappear.
Use distillation as an evaluated compression method.
Validate on real tasks
Track accuracy, calibration, subgroup behavior, latency, memory, and throughput. A small average-score change can hide a large regression on rare or safety-critical cases.
Keep preprocessing aligned
Teacher and student must agree on tokenization, labels, augmentation, and logit semantics. Silent target misalignment produces confident training curves and the wrong behavior.
Separate deployment claims
Parameter compression does not guarantee proportional speedups. Measure the exported student on the intended hardware, batch regime, precision, and serving stack.