Loading…
Glossary term
A fine-tuning method that trains directly on pairs of preferred and rejected responses.
DPO trains a model straight from pairs of preferred and rejected answers, with no separate reward model and no reinforcement-learning loop. The loss pushes the model to make the preferred answer more likely relative to the rejected one. That makes it simpler and more stable than RLHF.
The DPO loss
import torch.nn.functional as F
margin = beta * ((pi_chosen - ref_chosen) - (pi_rejected - ref_rejected))
loss = -F.logsigmoid(margin).mean()It has become a popular, lower-cost way to align models with human preferences.