S3-12 · 进阶 · 信息与梯度捷径S3-12 · Advanced · Shortcut for information and gradients
残差连接 · 模块加工,也保留原件Residual Connection · Transform the Input, Keep the Original
output = input + module(input)
残差连接(Residual
Connection)把模块输入直接加回模块输出,为前向信息与反向梯度提供捷径。A residual connection adds a module's input directly back to its output, creating a shortcut for forward
information and backward gradients.
01 · 修改时别丢原稿01 · Keep the original while editing
一份文件要连改八次:每次都覆盖原稿,还是让“修改量”叠加回原稿?A file needs eight edits. Would you overwrite the original each time, or add each change back onto
it?
x+F
02开关旁路,让信息小球穿过深层网络Toggle the Bypass and Send a Signal Through Depth
x + F(x)
输出保真度Output fidelity
88%
输入特征仍可辨input features retained
梯度亮度Gradient strength
82%
反向可达信号backward signal
可用路径Available paths
2
主路 + 捷径main + shortcut
小球展示前向信息;反向红光展示梯度。两者共享旁路,但它们不是同一个物理对象。The ball represents forward information; the red glow represents backward gradients. They share the
shortcut but are not the same physical object.
03残差连接的 4 个零件Four Parts of a Residual Connection
keep · change · add
PART 01
输入 xInput x
进入模块前的完整表示。The full representation before the module.
PART 02
主模块 FMain module F
计算需要加到输入上的变化量。Computes the change to add to the input.
PART 03
恒等旁路Identity bypass
让 x 绕过主模块直达加法点。Carries x around the main module to the add point.
PART 04
相加输出Added output
保留 x,同时叠加 F(x) 的修改。Keeps x while applying the change F(x).
04最小机制:让模块只学“改多少”Minimal Mechanism: Let the Module Learn the Change
y = x + F(x)
y = x + F(x) ∂y / ∂x = 1 + ∂F / ∂x
前向的 x 与反向公式里的 1
都来自旁路。关闭旁路并加深网络,两个仪表会同步衰减。The forward x and the backward 1 both come from the shortcut. Disable it and increase depth to make both
meters decay.
复制输入Copy input主路、旁路各一份One copy per path
主路求变化Compute the changeF(x) 不必重写全部F(x) need not rewrite everything
相加再前进Add and continue原信息可跨过深层Original information can survive depth
05边界实验:八层全部只走主路Boundary Test: Force Eight Layers Through the Main Path
主动制造失败Create a failure
深层连续改写会吞掉原信息与梯度Repeated rewrites erode information and gradients
把深度拉到
8,再关闭所有旁路。观察输出保真度与反向梯度亮度同时跌落。Set depth to eight and disable every shortcut. Watch output fidelity and backward-gradient strength fall
together.
尚未执行。旁路正在保留输入与梯度路径。Not run yet. Shortcuts are preserving input and gradient paths.
✗ 残差连接会跳过模块计算。A residual connection skips module computation.
→ ✓ 主路照常计算,旁路结果在末端相加。The main path still computes; the shortcut is added at the end.
✗ 旁路只是为了推理更快。The shortcut exists only for faster inference.
→ ✓ 关键作用是保留信息与梯度通路。Its key role is preserving information and gradient paths.
✗ 相加等于把两份文本拼起来。Addition means concatenating two texts.
→ ✓ 它对同形状表示逐分量相加。It adds same-shaped representations element by element.
06 · 一句话带走06 · One line to keep
残差连接把输入原样加回模块输出,让深层网络既能学习修改量,也给信息和梯度留一条捷径;开关旁路即可验证。A residual connection adds the input back to the module output, letting deep networks learn changes while
preserving a shortcut for information and gradients.