Token · How Models Slice Text

01 · 先别背定义 01 · Start with the puzzle

同样一句话，只改一个字符，为什么模型的计费数量会突然跳变？ Why can changing one character suddenly change how much text a model counts?

02 实时切分实验台 Live Tokenization Bench

输入实时切分型 Live text input

只改这段文字 Change only this text

每次输入都会重新切分。点画布里的任意色块，查看它对应的原文、教学 ID 和 UTF-8 字节。 Every keystroke reruns the split. Select any block on the canvas to inspect its source text, teaching ID, and UTF-8 bytes.

输入文本 Input text 0

教学分词模拟，非真实模型输出 simulated for teaching — not real model output

字符 Characters

Unicode code points

Token

离散片段 discrete pieces

字节 Bytes

UTF-8

03 刚才发生了什么 What Just Happened

4 个零件 Four parts

PART 01

原始字符串 Source string

你输入的是连续字符，包括空格、数字、标点和 emoji。 Your input is a character stream containing spaces, numbers, punctuation, and emoji.

PART 02

边界规则 Boundary rules

分词器决定哪里切开；英文词可能完整保留，也可能拆成多个片段。 The tokenizer chooses each cut. An English word may stay whole or split into several pieces.

PART 03

Token ID

每个 token 会映射到词表中的整数索引，模型处理的是这些 ID。 Each token maps to an integer index in a vocabulary. Those IDs are what the model processes.

PART 04

总数 Total count

成本、上下文窗口和生成长度通常按 token 计，不按字数计。 Cost, context windows, and generation length are usually measured in tokens, not characters.

04 最小机制：文字先切片，再变成编号 Minimal Mechanism: Slice First, Then Number

encode(text)

text → [t₁, t₂, …, t_n] → [id₁, id₂, …, id_n]

上方实验中，n 就是实时 token 总数；改回原文即可复现原计数。 In the bench above, n is the live token count. Restore the original text to reproduce the original count.

改一个字符 Change one character 原字符串改变 The source string changes

边界重算 Boundaries rerun 片段可能合并或拆开 Pieces may merge or split

计数跳变 The count jumps n 不必等于字数或词数 n need not equal words or characters

05 边界实验：亲手打破“一个词就是一个 token” Boundary Test: Break “One Word Equals One Token”

必须制造一次失败 Create a failure

反例按钮 Counterexample button

点击后放入一个长英文词。先猜“1 个词 = 1 个 token”，再看画布实际切成几块。 Load one long English word. Predict “one word = one token,” then count the actual blocks on the canvas.

尚未执行。点击按钮，把直觉交给结果检验。 Not run yet. Use the button and let the result test your intuition.

✗ 一个汉字或英文词固定等于一个 token。One Chinese character or English word always equals one token. → ✓ 边界由具体分词器和词表决定。Boundaries depend on the specific tokenizer and vocabulary.

✗ 字数可以直接代替 token 数。Character count can stand in for token count. → ✓ 混合语言、数字和 emoji 会让两者明显分离。Mixed scripts, numbers, and emoji can separate the two sharply.

✗ Token 就是模型理解的完整意思。A token is a complete unit of meaning. → ✓ Token 只是离散输入片段，意义来自后续表示与上下文。A token is only a discrete input piece; meaning emerges from later representations and context.

06 · 一句话带走 06 · One line to keep

Token 是模型读写文本时使用的离散片段，不等同于一个字或一个词；你可以在上方改一个字符，直接检验边界和总数会不会变。 A token is a discrete piece a model uses to read and write text—not necessarily a character or word. Change one character above to test whether the boundaries and count move.