S2-14 · 进阶 · 3DS2-14 · Advanced · 3D

学习率 · 方向对了,步长也可能错Learning Rate · Right Direction, Wrong Step Size

η scales every negative-gradient update

学习率是每次沿下降方向移动的步长系数。太小会慢,合适会稳定入谷,太大则可能跨过谷底,在两侧来回爆跳。The learning rate scales each move along the descent direction. Too small is slow, a suitable rate settles into the basin, and too large can overshoot and bounce across it.

01 · 下楼梯的步幅01 · Your stride on stairs

明明每一步都朝下,为什么迈得太大仍会越过平台、摔到另一边?Every step points downward, so why can an oversized stride still leap across the landing?

02同曲面、同起点,三种步长并跑Same Surface and Start, Three Step Sizes

唯一变量:学习率 ηOne variable: learning rate η
青 · 过小Cyan · small
0.16
η × 0.26
橙 · 中间Amber · center
0.62
η
紫 · 过大Violet · large
5.89
η × 9.5
正在加载曲面;若提示持续显示,则需要 WebGL / three.js。Loading surface; if this remains visible, WebGL or three.js is unavailable.
比较步 0 · 同起点comparison step 0 · same start
三条轨迹 · 拖动旋转three trails · drag to orbit
小:稳但慢small: stable but slow
中:稳定入谷center: settles well
大:越谷或爆跳large: overshoots or bounces

03学习率只做一件事:缩放更新Learning Rate Does One Job: Scale the Update

same gradient · different distance
DIRECTION

方向相同Same direction

三球都沿当前负梯度。All follow the current negative gradient.

SCALE

距离不同Different distance

η 直接放大或缩小更新量。η scales the update amount.

RECOMPUTE

落点不同Different landing

下一步从不同位置重算梯度。Next gradients come from different positions.

TRAJECTORY

轨迹分叉Paths diverge

小差异累积成慢、稳或震荡。Small differences become slow, stable, or oscillating paths.

04最小机制:η 乘在梯度前Minimal Mechanism: η Multiplies the Gradient

Δθ = −η∇L
Δθ = −η∇L(θ)
η 太小:Δθ 很短;η 合适:持续下降;η 太大:一步跨过低谷,下一步梯度反向,再跨回来。Tiny η makes short updates; a suitable η keeps descending; excessive η crosses the basin, flips the next gradient, and crosses back.
改变 ηChange η梯度方向不变direction unchanged
更新距离改变Update distance changes落在不同坡面land on a different slope
训练轨迹改变Training path changes慢、稳或发散slow, stable, or divergent

05边界实验:把学习率推到最大Boundary Test: Max Out the Learning Rate

主动制造失败Create a failure

让紫球反复跨谷Make the violet ball cross the basin

方向计算没有错,失败来自每一步太长。观察紫色轨迹在谷两侧跳动,而青球仍慢慢前进。The direction is correct; failure comes from steps that are too long. Watch violet bounce while cyan still crawls.

尚未执行。Not run yet.
学习率越大,训练一定越快。A larger learning rate always trains faster. → ✓ 过大会震荡甚至发散。Too large can oscillate or diverge.
学习率改变下山方向。Learning rate changes the descent direction. → ✓ 它缩放梯度给出的更新。It scales the gradient update.
一个学习率适合所有曲面。One rate fits every surface. → ✓ 合适步长取决于地形与尺度。A suitable rate depends on geometry and scale.
06 · 一句话带走06 · One line to keep

学习率不决定往哪走,而决定沿负梯度一步走多远;太小慢,合适稳,太大越谷。Learning rate does not choose direction; it sets distance along the negative gradient: too small is slow, suitable is stable, too large overshoots.