Mixture of Experts

01 · 一栋楼有很多专科诊室 01 · A building full of specialist clinics

整栋楼有四百名专家，为什么一个病人不需要让四百人同时会诊？ If a building houses four hundred specialists, why should one patient not activate all four hundred at once?

02 token 进路由器，调 top-k 点亮专家门 Route a Token and Set How Many Expert Doors Light

稀疏激活 Sparse activation

状态 A Metric A

—

状态 B Metric B

—

状态 C Metric C

—

直接操作主变量，再拖回去验证一句话结论。所有数值均为固定种子的教学模拟。 Operate the main variable, then move it back to verify the one-line conclusion. All values are fixed-seed teaching simulations.

03 把变化拆成 4 个零件 Break the Change into Four Parts

Mixture of Experts

PART 01

Token Token

每个输入位置分别得到路由分数，不同 token 可走不同专家。 Each input position receives routing scores and may take different experts.

PART 02

路由器 Router

路由器给专家排序，并选出当前 token 的 top-k。 The router ranks experts and selects top-k for the current token.

PART 03

专家门 Expert doors

只有被选中的专家执行前馈变换，其余参数本次休眠。 Only selected experts run their feed-forward transform; the rest sleep for this token.

PART 04

负载 Load

总容量再大，路由不均仍会让热门专家拥堵、其他专家闲置。 Even with huge total capacity, uneven routing congests popular experts and idles others.

04 最小机制：总参数固定，激活参数随 top-k 变化 Minimal Mechanism: Fixed Total Parameters, Active Parameters Follow Top-k

Mixture of Experts

y = Σ_i∈top-k g_i(x)E_i(x)

路由权重 g 只组合被选中的专家输出；未激活专家仍计入总参数，却不参与当前 token 计算。 Routing weights g combine only selected expert outputs. Inactive experts count toward total parameters but not this token's compute.

token 到达 Token arrives 路由器计算门分 router scores doors

选择 top-k Select top-k 少数专家激活 a few experts activate

合并输出 Combine outputs 容量大但计算稀疏 large capacity, sparse compute

05 边界实验：把相似 token 全挤进一扇门 Boundary Test: Funnel Similar Tokens into One Door

主动制造失败 Create a failure

总容量够大，局部仍会堵车 Plenty of total capacity, yet one route jams

把 top-k 设为 1，并制造偏置路由。8 个 token 都挤向同一专家，观察最忙负载达到 8/8。 Set top-k to 1 and create biased routing. All eight tokens crowd one expert and drive its load to 8/8.

尚未执行。先在主交互中观察正常机制。 Not run yet. Observe the normal mechanism in the main interaction first.

✗ 400M 总参数会为每个 token 全部运行。 All 400M parameters run for every token. → ✓ MoE 每次只激活 top-k 专家和共享部分。 MoE activates only top-k experts plus shared parts.

✗ 专家一定对应人类可命名学科。 Experts always map to human-named subjects. → ✓ 路由分工是训练形成的计算模式，未必可直接命名。 Routing specialization is a learned compute pattern and may not be directly nameable.

✗ top-k 越大一定越好。 Larger top-k is always better. → ✓ 更多激活提高计算与通信，也可能加重负载。 More activation raises compute and communication and may worsen load.

06 · 一句话带走 06 · One line to keep

MoE 用路由器让每个 token 只激活 top-k 专家；调 k 可验证总参数不动，而激活参数与负载一起改变。 MoE routes each token to only top-k experts. Change k to verify that total parameters stay fixed while active parameters and load change together.