Next-Token Prediction

01 · 先玩接话游戏 01 · Start with sentence completion

别人只说“天空是”，你脑中会浮出一个唯一答案，还是几个大小不同的候选？ When someone says “the sky is,” does one guaranteed ending appear—or several candidates with different strengths?

02 候选云：点一个 token，立刻进入下一轮 Candidate Cloud: Pick a Token, Trigger the Next Round

分数 → 概率 → 选字 · 3/3 score → probability → token · 3/3

预测轮次 Prediction round

每点一次 +1 +1 per pick

候选数 Candidates

玩具词表 toy vocabulary

最高概率 Top probability

—

03 一次“接下去”由 4 个零件组成 Four Parts of One “What Comes Next?”

逐 token token by token

PART 01

Context

输入框与已经点过的 token 共同形成当前上下文。 The input and every token already picked form the current context.

PART 02

Vocabulary

词表提供所有可被打分的下一 token。 The vocabulary supplies all next-token options that can receive scores.

PART 03

Distribution

logits 经 softmax 变成云中大小不同的概率。 Softmax turns logits into differently sized probabilities in the cloud.

PART 04

Selection

一次只选一个 token，它随后成为下一轮上下文。 One token is selected, then becomes context for the next round.

04 最小机制：概率回答“可能接什么”，不回答“事实是什么” Minimal Mechanism: Probability Predicts Text, Not Truth

P(token_t+1 | token_≤t)

P(x_t+1 | x₁…x_t)

条件线右侧是已经出现的 token，左侧是下一 token。点进序列后，条件立即改变。 Tokens to the right of the bar are existing context; the left side is the next token. Once picked, the condition changes immediately.

已有 token 改变 Context changes 条件输入更新 condition updates

重新打分 Scores recompute 新 logits 出现 new logits appear

新候选云 New cloud 下一轮再选一次 pick again next round

05 边界实验：让模型预测彩票号码 Boundary Test: Predict a Lottery Number

主动制造失败 Create a failure

流畅的候选分布不是事实来源 A fluent distribution is not a source of facts

把上下文换成“明天的彩票号码是”。页面仍会给候选概率，但这些数值不掌握未来开奖。 Change the context to “tomorrow's lottery number is.” The page still assigns probabilities, but those values do not know the future draw.

尚未执行。最高概率只表示文本候选排序。 Not run yet. Top probability only ranks text candidates.

✗ 最高概率 token 一定正确。 The highest-probability token must be correct. → ✓ 它只是在当前上下文下最可能续写。 It is only the most likely continuation in this context.

✗ 模型先想好整句，再逐字显示。 The model plans the whole sentence, then reveals it. → ✓ 每加入一个 token 都会重算下一轮。 Every appended token triggers a fresh prediction round.

✗ 概率就是模型的置信真值。 Probability is the model's truth confidence. → ✓ 这里的概率只描述候选 token 分布。 Here, probability describes only the token distribution.

06 · 一句话带走 06 · One line to keep

下一个 Token 预测会根据已有 token 生成一片候选概率云；点进任何一个候选，序列和下一轮分布都会立即改变。 Next-token prediction creates a probability cloud from existing tokens; pick any candidate and both the sequence and the next distribution change immediately.