TOKEN
可用片段 Available piece
字符、子词、词与特殊标记都可以是词表项。 Characters, subwords, words, and special markers may all be entries.
01 · 想象拼字积木盒
01 · Imagine a box of spelling blocks
积木盒里没有“training”整块时，要拼多少小块；把整块加入后又会怎样？ How many pieces spell “training” when no whole block exists—and what changes when that block enters the box?
02 拖动词表大小 Change Vocabulary Size
唯一变量：|V|One variable: |V||V|
16
可用 token 数
available tokens
常见句 token
Common sentence
0
training data powers AI
嵌入表行
Embedding rows
16
|V| × d
常见句 Common sentence
罕见词 Rare word
03 词表的 4 个零件 Four Parts of a Vocabulary
piece · ID · size · tableID
整数索引Integer index
每个词表项拥有唯一位置和 Token ID。 Each entry owns one unique position and Token ID.
SIZE
词表大小 Vocabulary size
|V| 是所有可用 token 的总数，不是当前句子长度。 |V| counts all available tokens, not tokens in the current sentence.
EMBEDDING TABLE
嵌入表行 Embedding rows
每个 Token ID 需要一行可学习向量。 Every Token ID needs one learnable vector row.
04 最小机制：分词器只能选词表已有片段 Minimal Mechanism: A Tokenizer Can Only Choose Listed Pieces
token ∈ V V = {token0, …, token
|V|−1}
E ∈ ℝ|V|×d
E ∈ ℝ|V|×d
扩大 V 可能减少常见文本的 token 数，却线性增加嵌入表行数。罕见词是否受益取决于新增了哪些片段。
Expanding V may reduce token count for common text, while linearly increasing embedding rows. Rare words benefit only if relevant pieces were added.
扩大 |V|Increase |V|
加入高频组合
add frequent combinations
边界合并
Merge boundaries
常见词片段减少
common words need fewer pieces
表行增多
Table grows
|V| rows × d columns
05 边界实验：词表有“退款”不等于模型知道政策 Boundary Test: Having “Refund” in Vocabulary Is Not Knowing the Policy
主动制造误用 Create a misuse把“refund”加入大词表 Add “refund” to the large vocabulary
它只让字符串可被一个 token 表示；退款期限、例外和权限来自训练或外部证据，不在词表条目里。 It only lets the string use one token. Refund windows, exceptions, and authority come from training or evidence, not the vocabulary entry.
尚未执行。先观察词表大小怎样改变切分。
Not run yet. First observe how vocabulary size changes tokenization.
✗
词表越大，模型事实一定越多。
A larger vocabulary guarantees more factual knowledge.
→ ✓
词表定义可用片段，不是事实库。
A vocabulary defines pieces, not facts.
✗
扩大词表会让所有词都变成单 token。
Expanding vocabulary makes every word one token.
→ ✓
只对新增或可组合的片段有效。
Only added or composable pieces benefit.
✗
词表大小等于当前 token 数。
Vocabulary size equals current token count.
→ ✓
前者是全体集合，后者是一次编码结果。
One is the full set; the other is one encoding result.
06 · 一句话带走
06 · One line to keep