Tokenizer · AI.WIKI

01 · 同一句话，两把剪刀01 · One sentence, two pairs of scissors

文字完全没改，只换一把“剪刀”，为什么 token 数就不同？ Why can the token count change when the text stays identical and only the “scissors” change?

02两种分词器切同一句话 Tokenize the Same Sentence Two Ways

唯一变量：分词器 One variable: tokenizer

分词器 Tokenizer

词优先Word-first

encode + decode

Token

边界数量boundary count

可逆Round-trip

✓

decode(encode(text))

解码结果 Decoded result

03分词器的 4 个零件 Four Parts of a Tokenizer

text · rules · IDs · text

STRING

输入字符串Input string

连续字符包含空格、标点、多语言与 emoji。 A character stream includes spaces, punctuation, scripts, and emoji.

ENCODER

编码器 Encoder

依据词表与算法决定每个 token 边界。 Vocabulary and algorithm determine every token boundary.

RANGE

原字符范围 Source range

每个 token 可追溯到原文中的起止位置。 Each token traces back to a start and end in the source.

DECODER

解码器 Decoder

把 token 序列按规则拼回字符串。 It reconstructs a string from the token sequence.

04最小机制：编码与解码是同一组件的两面 Minimal Mechanism: Encoding and Decoding Are Two Directions

decode(encode(s)) = s

s → [t₁ , …, t_n] → [id₁, …, id_n] → s

合法分词器应能用 token ID 恢复原字节序列。上方切换规则，n 会变，但解码结果应保持原文。 A valid tokenizer should recover the original byte sequence from token IDs. Switching rules changes n above, while decoded text should remain identical.

切换规则 Switch rules词优先或子词优先 word-first or subword-first

边界重算Boundaries rerun token 数改变token count changes

解码校验Decode check 仍应还原原文source should be restored

05 边界实验：怪异切分不是模型在“想”Boundary Test: Strange Splits Are Not Model “Thought”

主动制造反例Create a counterexample

放入长词与复合 emojiLoad a long word and compound emoji

同一字符串会被两种分词器切成不同数量；变化发生在模型推理之前。 Two tokenizers split the same string into different counts; the change happens before model reasoning.

尚未执行。先在普通句子上切换两种规则。 Not run yet. Switch rules on the normal sentence first.

✗ 分词器就是模型推理过程。 The tokenizer is the model's reasoning process. → ✓ 它是模型前后的独立编码组件。 It is a separate encoding component around the model.

✗ 同一句话永远有固定 token 数。 A sentence has one universal token count. → ✓ 数量取决于具体分词器与词表。Count depends on the tokenizer and vocabulary.

✗ token 边界一定按词义切。 Token boundaries always follow meaning. → ✓ 高频统计、字节与词表都会影响边界。 Frequency statistics, bytes, and vocabulary all affect boundaries.

06 · 一句话带走06 · One line to keep

分词器独立执行字符串与 token 序列之间的编码和解码；切换分词器会改变边界与数量，却不代表模型推理改变。 A tokenizer independently encodes strings to token sequences and decodes them back. Switching tokenizers changes boundaries and counts, not model reasoning.