Causal Language Modeling

01 · 猜题时不能翻答案页01 · No peeking at the answer page

如果做完填空后才遮住右侧答案，分数再高还能说明会预测吗？If the answers are hidden only after the exercise, does a high score prove genuine prediction?

02拖动预测游标，右侧 token 始终遮住Move the Prediction Cursor While Future Tokens Stay Hidden

玩具序列 · 逐步预测toy sequence · next step

状态 AMetric A

—

状态 BMetric B

—

状态 CMetric C

—

所有概率、得分与输出均为固定种子的教学模拟，不代表真实模型。拖回主变量即可复核结论。All probabilities, scores, and outputs are fixed-seed teaching simulations, not real model output. Move the main variable back to verify the conclusion.

03把刚才的变化拆成 4 个零件Break the Change into Four Parts

Causal Language Modeling

PREFIX

可见前缀Visible prefix

游标左侧 token 可以参与预测。Tokens left of the cursor can inform the prediction.

MASK

因果遮罩Causal mask

游标右侧未来 token 全部不可见。Every future token right of the cursor is hidden.

TARGET

下一个 tokenNext-token target

每一步只把紧邻右侧 token 当目标。Each step targets only the immediate token to the right.

LOOP

训练—推理衔接Training-inference match

推理时也用已生成前缀继续向右生成。Inference also uses the generated prefix to continue rightward.

04最小机制：主交互能逐项验证Minimal Mechanism: Verify Each Link in the Interaction

MECHANISM

loss_t = −log p(token_t+1 | token_≤t)

游标位置 t 决定可见前缀；右侧遮罩不动摇，loss 只落在 t+1。Cursor position t defines the visible prefix. The future stays masked, and loss lands only at t+1.

读取左侧Read the left 形成前缀表示build prefix representation

遮住未来Hide the future 禁止答案泄漏prevent answer leakage

预测下一词Predict next token 计算单步 losscompute one-step loss

05揭开右侧未来，让训练偷看答案Reveal the Future and Let Training Peek

主动制造失败Create a failure

边界实验Boundary experiment

失败按钮故意移除因果遮罩。loss 看似下降，但模型直接看到了待预测 token。The failure button removes the causal mask. Loss appears lower only because the target token is visible.

尚未执行。先在主交互中观察正常机制。Not run yet. Observe the normal mechanism in the main interaction first.

✗ 训练时可以看完整句，推理时再遮。Training can see the whole sentence and mask only at inference. → ✓ 这会泄漏未来，训练任务与推理不一致。That leaks the future and mismatches training with inference.

✗ 因果指模型理解真实因果关系。Causal means the model understands real-world causality. → ✓ 这里指序列可见方向，不是因果推断。Here it means sequence visibility, not causal inference.

✗ 每一步同时预测右侧所有 token。Each step predicts every future token at once. → ✓ 单个游标位置只对应紧邻的下一 token 目标。One cursor position corresponds to the immediate next-token target.

06 · 一句话带走06 · One line to keep

因果语言建模用左侧前缀预测紧邻的下一 token；游标右侧必须遮住，否则训练就是偷看答案。Causal language modeling predicts the immediate next token from the left prefix. The future must stay hidden or training is peeking.