Knowledge Distillation

01 · 只告诉答案，还是说清为什么犹豫01 · Give the answer—or also reveal uncertainty?

老师说“这是猫”，与老师说“72% 猫、28% 狐狸”相比，学生学到的边界信息会一样吗？A teacher says “cat.” Another says “72% cat, 28% fox.” Do those targets teach the same decision boundary?

02切换硬标签与教师概率分布，比较学生决策边界Switch Hard Labels and Teacher Probabilities to Compare Student Boundaries

教师分布 vs 硬标签teacher distribution vs hard labels

状态 AMetric A

—

状态 BMetric B

—

状态 CMetric C

—

所有概率、得分与输出均为固定种子的教学模拟，不代表真实模型。拖回主变量即可复核结论。All probabilities, scores, and outputs are fixed-seed teaching simulations, not real model output. Move the main variable back to verify the conclusion.

03把刚才的变化拆成 4 个零件Break the Change into Four Parts

Knowledge Distillation

TEACHER

教师模型Teacher model

较大模型提供概率分布或中间表示。A larger model supplies probabilities or internal representations.

SOFT TARGET

软目标Soft target

非零次选概率表达类别之间的相似性。Nonzero alternatives encode similarity among classes.

STUDENT

学生模型Student model

较小模型用蒸馏损失学习教师行为。A smaller model learns teacher behavior through distillation loss.

BOUNDARY

决策边界Decision boundary

软信息可让边界附近的变化更平滑。Soft information can make boundary behavior smoother.

04最小机制：主交互能逐项验证Minimal Mechanism: Verify Each Link in the Interaction

MECHANISM

loss = α·loss(hard_label) + (1−α)·T²·KL(student_T ∥ teacher_T)

样本与学生体积固定，只换目标信号；边界曲线和玩具准确率因此改变。Samples and student size stay fixed while only the target signal changes, shifting the boundary and toy accuracy.

教师出分布Teacher predicts保留次选概率retain alternative probabilities

学生拟合Student matches硬标签 + 软目标hard label + soft target

压缩部署Deploy smaller以验收确认保真verify retained behavior

05丢掉教师分布，只留 1/0 硬标签Discard the Teacher Distribution and Keep Only 1/0 Labels

主动制造失败Create a failure

边界实验Boundary experiment

学生体积没有变化，但教师对边界样本的相似性信息被抹掉，决策边界更僵硬。Student size stays fixed, but teacher similarity around boundary samples disappears, producing a more brittle boundary.

尚未执行。先在主交互中观察正常机制。Not run yet. Observe the normal mechanism in the main interaction first.

✗ 蒸馏就是复制教师最终答案。Distillation copies only the teacher's final answer.→ ✓ 核心价值之一是学习软分布或中间表示。A core benefit is learning soft distributions or representations.

✗ 学生一定与教师同样准确。The student always matches teacher accuracy.→ ✓ 容量、数据与蒸馏设置会产生保真损失。Capacity, data, and setup can all reduce fidelity.

✗ 教师错了，学生会自动纠正。The student automatically corrects teacher errors.→ ✓ 教师偏差也可能被蒸馏进学生。Teacher bias can be distilled into the student too.

06 · 一句话带走06 · One line to keep

知识蒸馏让小学生模型学习教师的软分布或表示；它能传递类别相似性，但也会传递教师偏差并产生压缩损失。Knowledge distillation teaches a smaller student from teacher distributions or representations. It can transfer class similarity, along with teacher bias and some compression loss.