Byte Pair Encoding

01 · 从反复出现的拼图边开始 01 · Start with repeated puzzle edges

如果 l 后面总跟着 o，为什么不把 l+o 做成一块可重复使用的新积木？ If l is repeatedly followed by o, why not turn l+o into one reusable piece?

02 执行下一次最高频合并 Execute the Next Most-frequent Merge

唯一变量：合并轮次 One variable: merge round

合并轮次 Merge round

逐轮学习 learned stepwise

当前词表项 Vocabulary entries

字符 + 新合并 characters + merges

平均 token/词 Avg tokens/word

0.00

越合并通常越少 usually falls with merges

03 一轮 BPE 的 4 个零件 Four Parts of One BPE Round

count · choose · merge · repeat

CORPUS

训练语料 Training corpus

合并频率只来自当前提供的文本样本。 Merge frequencies come only from the provided text samples.

PAIR COUNT

相邻对计数Adjacent-pair count

只统计紧挨的两个片段，不看整词语义。 Only adjacent pieces count; whole-word meaning is not inspected.

MERGE

全局合并 Global merge

最高频对的所有出现位置同时变成新片段。 Every occurrence of the winning pair becomes one new piece.

VOCABULARY

词表增长Vocabulary growth

新合并片段加入词表，供后续编码使用。 The merged piece enters the vocabulary for future encoding.

04 最小机制：每轮贪心合并最高频相邻对 Minimal Mechanism: Greedily Merge the Most Frequent Pair Each Round

argmax count(a,b)

(a*, b*) = arg max_(a,b) count(a,b)
[…,a*,b*,…] → […,a*b*,…]

按钮每点一次，只执行一轮；合并后重新计数，因为新的邻接关系已经出现。 Each click runs exactly one round. Counts are recomputed afterward because new adjacencies have appeared.

统计相邻对 Count adjacent pairs l+o, o+w, w+</w>…

选择最高频 Choose the winner并列时固定次序 fixed tie order

全部合并 Merge everywhere 词表增一项，再循环 add one entry, repeat

05 边界实验：高频片段不等于语义单元 Boundary Test: A Frequent Piece Is Not Necessarily a Semantic Unit

主动制造反例 Create a counterexample

自动跑 6 轮，再测试 unseen “newer” Run six rounds, then test unseen “newer”

BPE 会优先适配 low/lower/lowest/slow 的统计。新词不一定受益，合并片段也不保证对应完整词义。 BPE prioritizes statistics from low/lower/lowest/slow. An unseen word may not benefit, and a merged piece need not map to a semantic unit.

尚未执行。先手动完成一两轮。 Not run yet. Complete one or two rounds manually first.

✗ BPE 每次合并一个完整词义。 Every BPE merge captures one complete meaning. → ✓ 它按相邻频率合并片段。 It merges pieces by adjacency frequency.

✗ 训练语料高频就对所有文本高频。 Frequent in training means frequent everywhere. → ✓ 语料分布决定合并偏好。 Corpus distribution determines merge preference.

✗ 合并越多就没有代价。 More merges have no cost. → ✓ 每次合并都会扩大词表与嵌入表。 Every merge expands vocabulary and embedding tables.

06 · 一句话带走 06 · One line to keep

BPE 从字符或字节出发，反复合并语料中最高频的相邻片段来学习 token；逐轮点击即可检验词表增长与平均 token 数下降。 BPE starts from characters or bytes and repeatedly merges the most frequent adjacent pieces to learn tokens. Step through rounds to verify vocabulary growth and average token-count reduction.