Loading…
Glossary term
Training a smaller model to imitate a larger one, keeping much of its ability at lower cost.
In distillation, a small 'student' model is trained to imitate a large 'teacher'. Instead of only learning from hard labels, the student matches the teacher's full probability distribution, which carries richer information.
The result keeps much of the teacher's skill at a fraction of the size and cost.
The distillation loss
import torch.nn.functional as F
T = 2.0 # temperature softens the teacher's probabilities
loss = F.kl_div(
F.log_softmax(student_logits / T, dim=-1),
F.softmax(teacher_logits / T, dim=-1),
reduction="batchmean",
) * (T * T)It is a main way to make fast, cheap models for phones and high-volume services.