S3-12 · 进阶 · 信息与梯度捷径S3-12 · Advanced · Shortcut for information and gradients

残差连接 · 模块加工,也保留原件Residual Connection · Transform the Input, Keep the Original

output = input + module(input)

残差连接(Residual Connection)把模块输入直接加回模块输出,为前向信息与反向梯度提供捷径。A residual connection adds a module's input directly back to its output, creating a shortcut for forward information and backward gradients.

01 · 修改时别丢原稿01 · Keep the original while editing

一份文件要连改八次:每次都覆盖原稿,还是让“修改量”叠加回原稿?A file needs eight edits. Would you overwrite the original each time, or add each change back onto it?

02 开关旁路,让信息小球穿过深层网络Toggle the Bypass and Send a Signal Through Depth

x + F(x)
输出保真度Output fidelity
88%
输入特征仍可辨input features retained
梯度亮度Gradient strength
82%
反向可达信号backward signal
可用路径Available paths
2
主路 + 捷径main + shortcut

小球展示前向信息;反向红光展示梯度。两者共享旁路,但它们不是同一个物理对象。The ball represents forward information; the red glow represents backward gradients. They share the shortcut but are not the same physical object.

03 残差连接的 4 个零件Four Parts of a Residual Connection

keep · change · add
PART 01

输入 xInput x

进入模块前的完整表示。The full representation before the module.

PART 02

主模块 FMain module F

计算需要加到输入上的变化量。Computes the change to add to the input.

PART 03

恒等旁路Identity bypass

让 x 绕过主模块直达加法点。Carries x around the main module to the add point.

PART 04

相加输出Added output

保留 x,同时叠加 F(x) 的修改。Keeps x while applying the change F(x).

04 最小机制:让模块只学“改多少”Minimal Mechanism: Let the Module Learn the Change

y = x + F(x)
y = x + F(x)
∂y / ∂x = 1 + ∂F / ∂x
前向的 x 与反向公式里的 1 都来自旁路。关闭旁路并加深网络,两个仪表会同步衰减。The forward x and the backward 1 both come from the shortcut. Disable it and increase depth to make both meters decay.
复制输入Copy input主路、旁路各一份One copy per path
主路求变化Compute the changeF(x) 不必重写全部F(x) need not rewrite everything
相加再前进Add and continue原信息可跨过深层Original information can survive depth

05 边界实验:八层全部只走主路Boundary Test: Force Eight Layers Through the Main Path

主动制造失败Create a failure

深层连续改写会吞掉原信息与梯度Repeated rewrites erode information and gradients

把深度拉到 8,再关闭所有旁路。观察输出保真度与反向梯度亮度同时跌落。Set depth to eight and disable every shortcut. Watch output fidelity and backward-gradient strength fall together.

尚未执行。旁路正在保留输入与梯度路径。Not run yet. Shortcuts are preserving input and gradient paths.
残差连接会跳过模块计算。A residual connection skips module computation. → ✓ 主路照常计算,旁路结果在末端相加。The main path still computes; the shortcut is added at the end.
旁路只是为了推理更快。The shortcut exists only for faster inference. → ✓ 关键作用是保留信息与梯度通路。Its key role is preserving information and gradient paths.
相加等于把两份文本拼起来。Addition means concatenating two texts. → ✓ 它对同形状表示逐分量相加。It adds same-shaped representations element by element.
06 · 一句话带走06 · One line to keep

残差连接把输入原样加回模块输出,让深层网络既能学习修改量,也给信息和梯度留一条捷径;开关旁路即可验证。A residual connection adds the input back to the module output, letting deep networks learn changes while preserving a shortcut for information and gradients.