Evaluation Contamination

01 · 把考试题贴进复习提纲，再考高分是在测能力还是记答案01 · If exam questions enter the study guide, are high scores measuring ability or recall?

一条留出题被复制进提示库，另一条答案被知识库检索命中。分数上涨了 11 分——这还能叫模型泛化提升吗？One held-out case is copied into the prompt library and another answer becomes retrievable from the knowledge base. The score rises eleven points. Is that still evidence of better generalization?

02把测试样本送入提示库或知识库，观察血缘线变红并让指标失信Route Test Cases into the Prompt or Knowledge Library and Watch Lineage Turn Red and Metrics Lose Trust

评测证据血缘 · 蓝/黄/红eval lineage · blue/yellow/red

状态 AMetric A

—

状态 BMetric B

—

状态 CMetric C

—

所有概率、分数、成本、token 与输出均为固定种子的教学模拟，不代表真实模型输出。把主变量拖回即可复核结论。All probabilities, scores, costs, tokens, and outputs are fixed-seed teaching simulations, not real model output. Move the main variable back to verify the conclusion.

03把刚才的变化拆成 4 个零件Break the Change into Four Parts

Evaluation Contamination

SOURCE

测试来源Test source

每条题与答案都有不可混淆的来源 ID。Every question and answer carries an unambiguous source ID.

PATH

开发路径Development paths

训练、提示、检索与调参是四类可能交叉点。Training, prompting, retrieval, and tuning are four possible intersections.

DETECT

血缘检测Lineage detection

同源样本进入开发链，路径立即从蓝变红。When the same source enters development, the path turns from blue to red.

TRUST

指标可信状态Metric trust state

受影响的分数保留数值但标“不可采信”，不得用于发布门。Keep the number for audit but mark it untrustworthy and exclude it from release gates.

04最小机制：主交互能逐项验证Minimal Mechanism: Verify Each Link in the Interaction

MECHANISM

trustworthy(metric) = no(test_lineage ∩ development_lineage)

只要测试血缘与提示、知识库、训练或调参任一路径相交，受影响指标就必须标不可采信。If test lineage intersects prompting, retrieval, training, or tuning anywhere, affected metrics must be marked untrustworthy.

追踪测试 IDTrack test IDs题与答案同源questions + answers

检测路径相交Detect intersection提示 / 知识 / 训练prompt / knowledge / training

降级证据Downgrade evidence指标不可采信metric not trustworthy

05同时把一条测试题拖入提示库和知识库Send One Test Case into Both Prompt and Knowledge Libraries

主动制造失败Create a failure

边界实验Boundary experiment

分数可以仍然显示，但必须保留红色血缘并退出发布判断。清空界面不能洗掉已发生的污染。The score may remain visible, but its red lineage must remain and it must leave release decisions. Clearing the UI cannot erase contamination history.

尚未执行。先在主交互中观察正常机制。Not run yet. Observe the normal mechanism in the main interaction first.

✗ 只要不是训练污染就没关系。Only training contamination matters.→ ✓ 提示、检索与调参同样会让答案提前可见。Prompting, retrieval, and tuning can expose answers too.

✗ 重新跑一次同样测试就恢复可信。Rerunning the same test restores trust.→ ✓ 必须换真正未见的新留出集并隔离开发路径。Use a genuinely unseen held-out set and isolate development paths.

✗ 污染分数删掉就不必留记录。Deleting the contaminated score removes the issue.→ ✓ 保留数值和红色血缘用于审计，但不用于结论。Retain the value and red lineage for audit, but exclude them from conclusions.

06 · 一句话带走06 · One line to keep

评测污染发生在测试血缘与训练、提示、检索或调参链相交时；受影响分数可以保留审计，但必须红标为不可采信。Evaluation contamination begins when test lineage intersects training, prompting, retrieval, or tuning. Keep affected scores for audit, but mark them red and untrustworthy.