Pairs where the chosen response has the larger implicit reward.
DPO trains directly on preferred and rejected response pairs while anchoring the policy to a reference model. Tune beta, preference strength, and label noise, then watch likelihood margins and policy drift move together.
Open preference surfaceA chosen response is not assigned an absolute score. It only needs a larger policy-versus-reference log-ratio than the rejected response. DPO turns that pairwise comparison into a logistic classification objective without explicitly fitting a reward model or running on-policy PPO rollouts.
Pairs where the chosen response has the larger implicit reward.
Chosen minus rejected policy/reference log-ratio.
Negative log-sigmoid of the beta-scaled margin.
Modeled distance from the frozen reference policy.
Each response is scored by how much more or less likely it is under the trainable policy than under the frozen reference. This prevents preference training from merely favoring generally common text.
r̂(x,y) = β log(πθ(y|x) / πref(y|x))The logistic loss rewards a positive difference between the chosen and rejected implicit rewards. Ambiguous pairs stay near the decision boundary and create the strongest gradient.
L = −log σ(r̂chosen − r̂rejected)Beta sets the relationship between preference margin and reference anchoring. Dataset quality still dominates: contradictory or stylistic shortcuts can train the wrong behavior very efficiently.
preference gain ↔ policy drift tradeoffThe objective may push larger policy changes to separate pairs, increasing preference fit and possible capability drift.
Monitor held-out behavior.Incorrect chosen/rejected labels reverse the gradient. High-confidence optimization can amplify annotation mistakes.
Audit disagreements and annotator policy.The model can learn verbosity, formatting, or hedging that correlates with labels without improving underlying correctness.
Evaluate dimensions separately.