LLM-as-a-Judge · AI.WIKI

01 · 同两份答案只换左右，冠军也跟着换，裁判在评什么01 · If the winner flips when two answers swap sides, what is the judge measuring?

候选身份已经隐藏，但 A 放左边得 8.4 分，交换到右边只剩 7.7 分。你会把第一次结果直接扩展到十万条吗？Candidate identity is hidden, yet A scores 8.4 on the left and 7.7 on the right. Would you scale the first result to one hundred thousand cases?

02隐藏候选身份并交换顺序，检测位置偏差，再用人工小样本校准Blind Candidate Identity, Swap Order, Detect Position Bias, and Calibrate on a Human Sample

评测证据血缘 · 蓝/黄/红eval lineage · blue/yellow/red

状态 AMetric A

—

状态 BMetric B

—

状态 CMetric C

—

所有概率、分数、成本、token 与输出均为固定种子的教学模拟，不代表真实模型输出。把主变量拖回即可复核结论。All probabilities, scores, costs, tokens, and outputs are fixed-seed teaching simulations, not real model output. Move the main variable back to verify the conclusion.

03把刚才的变化拆成 4 个零件Break the Change into Four Parts

LLM-as-a-Judge

RUBRIC

明确量规Explicit rubric

把事实、完整性与风格拆成可复核维度。Split factuality, completeness, and style into reviewable dimensions.

BLIND

身份盲化Identity blinding

移除模型名、厂商与版本暗示。Remove model names, vendors, and version cues.

SWAP

顺序互换Order swap

同一对答案左右互换后重复判定。Repeat judgment after swapping the same pair left to right.

CAL

人工校准Human calibration

用小样本人评估计一致率与系统偏差。Use a small human-rated sample to estimate agreement and systematic bias.

04最小机制：主交互能逐项验证Minimal Mechanism: Verify Each Link in the Interaction

MECHANISM

position_bias = |score(A_left) − score(A_right)|

主交互在换位后直接比较同一答案的分差；超过阈值就亮位置偏差，并显示人工样本一致率。After swapping, the interaction compares the same answer directly. A gap above threshold triggers the position-bias alert alongside human agreement.

盲化候选Blind candidates移除身份线索remove identity cues

交换顺序Swap order同题重判rejudge same pair

人工校准Calibrate估计一致与偏差estimate agreement + bias

05关闭人工校准，只保留单次左优先判分Disable Human Calibration and Keep One Left-first Judgment

主动制造失败Create a failure

边界实验Boundary experiment

规模化不能修复系统偏差，只会稳定地复制它。必须先换位复测并用人评小样本量裁判。Scale does not repair systematic bias; it reproduces it consistently. Swap-test and calibrate against a human sample first.

尚未执行。先在主交互中观察正常机制。Not run yet. Observe the normal mechanism in the main interaction first.

✗ 隐藏模型名就已经完全公平。Hiding model names makes judging fully fair.→ ✓ 仍需交换顺序检测位置效应。Order swaps are still required to detect position effects.

✗ 裁判与人类一次一致就校准完成。One agreement with a human completes calibration.→ ✓ 用覆盖关键边界的小样本估计一致率与分层偏差。Estimate agreement and subgroup bias on a boundary-covering sample.

✗ 裁判分数是客观真值。Judge scores are objective truth.→ ✓ 它是带误差的量具，必须保留量规、版本和校准记录。It is a noisy instrument; retain rubric, version, and calibration records.

06 · 一句话带走06 · One line to keep

模型裁判是可扩展但有偏的量具；隐藏身份、交换顺序与人工小样本校准缺一不可，换位变分就必须亮位置偏差。An LLM judge is scalable but biased. Identity blinding, order swaps, and human calibration are all required; score movement after a swap must trigger a position-bias warning.