DPO · AI.WIKI

01 · 同一张意见表，能否少转两道手01 · Can the same preference form skip two handoffs?

既然已经知道回答 A 优于 B，能否直接让模型提高 A 相对 B 的概率，而不先训练打分员再跑强化学习？If response A is already preferred to B, can the policy raise A relative to B directly instead of training a scorer and then running RL?

02把 RLHF 与 DPO 流程并排，拖同一偏好对的间隔看更新方向Compare RLHF and DPO Side by Side Using the Same Preference Pair

并排流程 · 同对更新side-by-side pipelines · same pair

状态 AMetric A

—

状态 BMetric B

—

状态 CMetric C

—

所有概率、得分与输出均为固定种子的教学模拟，不代表真实模型。拖回主变量即可复核结论。All probabilities, scores, and outputs are fixed-seed teaching simulations, not real model output. Move the main variable back to verify the conclusion.

03把刚才的变化拆成 4 个零件Break the Change into Four Parts

Direct Preference Optimization

PAIR

优劣回答对Chosen/rejected pair

同一输入下明确哪份回答更优。Identify the preferred response for the same prompt.

REFERENCE

参考策略Reference policy

更新相对基线受控，避免无界漂移。Updates stay relative to a baseline instead of drifting without bound.

DIRECT LOSS

直接损失Direct loss

直接比较策略对优选与拒选的相对概率。Compare the policy's relative likelihood for chosen and rejected responses.

POLICY

策略更新Policy update

不经过独立奖励模型即可推动偏好方向。Move the policy toward preference without a separate reward model.

04最小机制：主交互能逐项验证Minimal Mechanism: Verify Each Link in the Interaction

MECHANISM

loss_DPO ∝ −log σ(β[(log π(y⁺)−log π(y⁻))−reference])

拖动 preference margin，DPO 的更新箭头同步变长；RLHF 与 DPO 流程始终并排，节点差异不会被切换动画藏住。Moving preference margin changes the DPO update arrow. RLHF and DPO remain side by side so their structural differences are never hidden by a toggle.

收集偏好对Collect pairschosen / rejectedchosen / rejected

比较相对概率Compare likelihoods策略相对参考policy relative to reference

直接更新Update directly不单训奖励模型no separate reward model

05把优选与拒选打成平手Make Chosen and Rejected Responses a Tie

主动制造失败Create a failure

边界实验Boundary experiment

DPO 省掉流程节点，不会凭空制造偏好信号。间隔为 0 时，直接更新只会追逐标注噪声。DPO removes pipeline stages; it does not create preference signal from nothing. At zero margin, direct updates chase annotation noise.

尚未执行。先在主交互中观察正常机制。Not run yet. Observe the normal mechanism in the main interaction first.

✗ DPO 与 RLHF 完全相同，只是名字更短。DPO is identical to RLHF with a shorter name.→ ✓ DPO 不需要独立奖励模型或在线强化学习。DPO does not require a separate reward model or online RL.

✗ 流程更短就一定更稳定。A shorter pipeline is always more stable.→ ✓ 稳定性仍依赖偏好对、β 与参考策略。Stability still depends on pairs, β, and the reference policy.

✗ DPO 不需要拒选回答。DPO needs no rejected response.→ ✓ 相对比较需要 chosen 与 rejected 两端。Relative comparison needs both chosen and rejected responses.

06 · 一句话带走06 · One line to keep

DPO 用同一优劣回答对直接调整策略的相对概率；它绕过奖励模型与在线强化学习，但仍受偏好对和参考策略约束。DPO uses the same chosen/rejected pair to adjust relative policy likelihood directly. It bypasses a reward model and online RL, while still depending on pair quality and a reference policy.