RLHF · AI.WIKI

01 · 流水线末端的偏差从哪来01 · Where did the final bias enter?

最终模型只偏爱一种写法，是策略更新出了错，还是更早的候选、标注或奖励模型已经偏了？The final policy favors one writing style. Did policy optimization cause it, or were sampling, ranking, or reward fitting already skewed?

02在四站任一处注入偏斜，看它沿箭头传到最终行为Inject Skew at Any of Four Stages and Watch It Reach Final Behavior

四站流水线 · 偏斜传导four-stage pipeline · skew propagation

状态 AMetric A

—

状态 BMetric B

—

状态 CMetric C

—

所有概率、得分与输出均为固定种子的教学模拟，不代表真实模型。拖回主变量即可复核结论。All probabilities, scores, and outputs are fixed-seed teaching simulations, not real model output. Move the main variable back to verify the conclusion.

03把刚才的变化拆成 4 个零件Break the Change into Four Parts

Reinforcement Learning from Human Feedback

SAMPLE

采样候选Sample candidates

先决定人类能看到哪些回答。First determine which responses people can see.

RANK

人类排序Human ranking

标注者按明示准则比较候选。Annotators compare candidates against explicit criteria.

REWARD

奖励拟合Reward fitting

偏好对被压成可用于优化的代理分。Preference pairs become a proxy score for optimization.

POLICY

策略更新Policy update

模型被推向高奖励行为，同时受约束防止漂移。The policy moves toward high reward while constraints limit drift.

04最小机制：主交互能逐项验证Minimal Mechanism: Verify Each Link in the Interaction

MECHANISM

final_shift = sample_bias → rank_bias → reward_bias → policy_update

点击的站点是偏斜起点；它及后续站点都会变红，明确“末端问题可能源自上游”。The selected stage is the skew origin. It and every downstream stage turn red, making upstream causes of final behavior visible.

收集比较Collect comparisons采样 + 人类排序sampling + human ranking

学习代理Learn proxy奖励模型拟合fit a reward model

更新策略Update policy优化并持续验收optimize and keep evaluating

05只让单一偏好群体完成第 2 站排序Let One Narrow Preference Group Own Stage 2 Rankings

主动制造失败Create a failure

边界实验Boundary experiment

奖励模型不会自动补回未出现的偏好；排序站偏斜会被后两站继续学习与放大。The reward model cannot restore preferences it never saw. Skew in the ranking stage is learned and amplified downstream.

尚未执行。先在主交互中观察正常机制。Not run yet. Observe the normal mechanism in the main interaction first.

✗ RLHF 就是人工给最终模型点赞。RLHF is simply humans liking the final model.→ ✓ 它是多阶段数据与优化流水线。It is a multi-stage data and optimization pipeline.

✗ 最后行为偏了，只需重跑策略更新。If behavior is skewed, rerun policy optimization only.→ ✓ 必须定位偏斜从哪一站进入。First locate the stage where skew entered.

✗ 有奖励模型就不再需要人工。A reward model eliminates further human input.→ ✓ 奖励代理仍要持续用人类与真实结果校准。The reward proxy still needs calibration against people and real outcomes.

06 · 一句话带走06 · One line to keep

RLHF 是四站串联的反馈流水线；任何一站的样本或准则偏斜，都会沿奖励信号传到最终策略。RLHF is a four-stage feedback pipeline. Sampling or criteria skew at any stage can propagate through the reward signal into the final policy.