Model Quantization

01 · 细刻度尺换成粗刻度尺01 · Replace a fine ruler with a coarse one

把每个权重从 16 位刻度压成 4 位，文件会更小；原来落在两条细刻度之间的数要被推到哪里？Compress every weight from 16-bit to 4-bit and the file shrinks. But where do values between the remaining coarse levels go?

02从 16 位切到 8 位、4 位，看大小与延迟下降、误差与样例差异上升Move from 16 to 8 to 4 Bits and Compare Size, Latency, Error, and Output Drift

精度—成本权衡precision-cost trade-off

状态 AMetric A

—

状态 BMetric B

—

状态 CMetric C

—

所有概率、得分与输出均为固定种子的教学模拟，不代表真实模型。拖回主变量即可复核结论。All probabilities, scores, and outputs are fixed-seed teaching simulations, not real model output. Move the main variable back to verify the conclusion.

03把刚才的变化拆成 4 个零件Break the Change into Four Parts

Model Quantization

RANGE

数值范围Numeric range

量化前先确定要覆盖的最小与最大值。First define the minimum and maximum values to represent.

LEVELS

离散层级Discrete levels

位宽决定可用刻度数量。Bit width determines the number of representable levels.

ROUND

舍入映射Rounding map

原值被放到最近可用层级。Each original value maps to a nearby available level.

TRADE-OFF

部署权衡Deployment trade-off

文件与延迟下降，误差与漂移风险上升。File size and latency fall as error and drift risk rise.

04最小机制：主交互能逐项验证Minimal Mechanism: Verify Each Link in the Interaction

MECHANISM

q(w) = round((w−min)/scale)·scale + min

灰线固定原权重，红线显示当前量化值；切位宽直接改变层级数量与两者距离。Gray lines keep original weights fixed; red lines show current quantized values. Bit width directly changes level count and their distance.

选择位宽Choose precision16 / 8 / 4-bit16 / 8 / 4-bit

映射刻度Map to levels原值舍入到离散层round values to discrete levels

部署验收Evaluate deployment大小、延迟、误差、样例size, latency, error, examples

05不做校准，直接压到 4 位Jump Straight to 4 Bits Without Calibration

主动制造失败Create a failure

边界实验Boundary experiment

离群权重被粗刻度拉偏，固定 5 个样例已有 3 个输出改变；资源收益不能替代质量验收。Coarse levels distort outlier weights and 3 of 5 fixed examples change. Resource savings do not replace quality evaluation.

尚未执行。先在主交互中观察正常机制。Not run yet. Observe the normal mechanism in the main interaction first.

✗ 位宽减半，误差也必然减半。Halving bit width halves error.→ ✓ 层级数量按 2 的幂变化，误差依赖分布与校准。Level count changes exponentially, while error depends on distribution and calibration.

✗ 文件变小就说明模型更快。A smaller file guarantees faster inference.→ ✓ 硬件内核、带宽与反量化也影响延迟。Hardware kernels, bandwidth, and dequantization also affect latency.

✗ 量化只改文件，不会改输出。Quantization changes only the file, never outputs.→ ✓ 舍入误差可能累积成样例差异。Rounding error can accumulate into output drift.

06 · 一句话带走06 · One line to keep

模型量化用更少位数换取更小文件与潜在低延迟；刻度变粗会增加舍入误差，因此必须连同固定样例一起验收。Model quantization trades fewer bits for smaller files and potentially lower latency. Coarser levels increase rounding error, so fixed examples must be evaluated alongside resource metrics.