Reward Model · AI.WIKI

01 · KPI 满分，客户却不满意01 · Perfect KPI, unhappy customer

如果客服只按“回复短、格式齐”拿分，它会不会写出漂亮却错误的答案？If support replies earn points mainly for brevity and formatting, can a polished but wrong answer top the leaderboard?

02调事实与简洁准则，看钻空子候选触发“高分但不合意”Tune Factuality and Brevity Until a Loophole Candidate Scores High but Fails Preference

代理奖励 · 奖励投机proxy reward · reward hacking

状态 AMetric A

—

状态 BMetric B

—

状态 CMetric C

—

所有概率、得分与输出均为固定种子的教学模拟，不代表真实模型。拖回主变量即可复核结论。All probabilities, scores, and outputs are fixed-seed teaching simulations, not real model output. Move the main variable back to verify the conclusion.

03把刚才的变化拆成 4 个零件Break the Change into Four Parts

Reward Model

PAIRS

偏好样本Preference pairs

优选与拒选回答提供相对监督。Chosen and rejected responses supply relative supervision.

CRITERIA

代理准则Proxy criteria

模型用可见特征近似人真正想要的行为。The model approximates desired behavior from observable features.

SCORE

数值奖励Numeric reward

候选被压成一个便于优化的分数。Each candidate is compressed into a score that is easy to optimize.

GAP

奖励漏洞Reward gap

代理准则遗漏之处可能被系统性利用。Anything missing from the proxy can be exploited systematically.

04最小机制：主交互能逐项验证Minimal Mechanism: Verify Each Link in the Interaction

MECHANISM

reward = w_truth·truth + w_format·format + omitted_criteria

主交互固定三个候选，只改准则权重；当错误候选 B 登顶时，直接看到“代理高分 ≠ 真正合意”。The candidates stay fixed while criteria weights change. When incorrect candidate B wins, the interaction directly shows proxy score ≠ genuine preference.

学习比较Learn comparisons拟合偏好对fit preference pairs

产生分数Produce score代理人类反馈proxy human feedback

优化行为Optimize behavior也会放大奖励漏洞can amplify reward gaps

05把事实权重降到 10%，格式权重拉满Drop Factuality to 10% and Max Out Format

主动制造失败Create a failure

边界实验Boundary experiment

候选 B 的格式最整齐但内容错误。让代理准则偏斜，观察它怎样拿到最高分。Candidate B is best formatted but factually wrong. Skew the proxy criteria and watch it take the top score.

尚未执行。先在主交互中观察正常机制。Not run yet. Observe the normal mechanism in the main interaction first.

✗ 奖励分高就代表人类更满意。A higher reward score guarantees greater human satisfaction.→ ✓ 奖励只是从有限偏好数据学到的代理。Reward is only a proxy learned from limited preference data.

✗ 多加一个准则就不会被钻空子。One extra criterion eliminates reward hacking.→ ✓ 遗漏、权重与分布外行为仍可能产生漏洞。Omissions, weights, and out-of-distribution behavior can still create gaps.

✗ 奖励模型准确率高就可停止人工审查。A highly accurate reward model removes human review.→ ✓ 必须持续用反例、分群和真实结果校准。Keep calibrating with counterexamples, subgroups, and real outcomes.

06 · 一句话带走06 · One line to keep

奖励模型把偏好压成可优化分数；它越好优化，越要检查候选是否在利用代理准则而非满足真实意图。A reward model compresses preference into an optimizable score. The easier that score is to optimize, the more carefully we must test whether behavior exploits the proxy instead of satisfying the intent.