学习率 · 方向对了,步长也可能错Learning Rate · Right Direction, Wrong Step Size
η scales every negative-gradient update
学习率是每次沿下降方向移动的步长系数。太小会慢,合适会稳定入谷,太大则可能跨过谷底,在两侧来回爆跳。The learning rate scales each move along the descent direction. Too small is slow, a
suitable rate settles into the basin, and too large can overshoot and bounce across
it.
01 · 下楼梯的步幅01 · Your stride on stairs
明明每一步都朝下,为什么迈得太大仍会越过平台、摔到另一边?Every step points downward, so why can an oversized stride still leap across the
landing?
η×
02同曲面、同起点,三种步长并跑Same Surface and Start, Three Step Sizes
唯一变量:学习率 ηOne variable: learning rate η
青 · 过小Cyan · small
0.16
η × 0.26
橙 · 中间Amber · center
0.62
η
紫 · 过大Violet · large
5.89
η × 9.5
正在加载曲面;若提示持续显示,则需要 WebGL /
three.js。Loading surface; if this remains visible, WebGL or three.js is unavailable.
比较步 0 · 同起点comparison step 0 · same start
三条轨迹 · 拖动旋转three trails · drag to orbit
小:稳但慢small: stable but slow
中:稳定入谷center: settles well
大:越谷或爆跳large: overshoots or bounces
03学习率只做一件事:缩放更新Learning Rate Does One Job: Scale the Update
same gradient · different distance
DIRECTION
方向相同Same direction
三球都沿当前负梯度。All follow the current negative gradient.
SCALE
距离不同Different distance
η 直接放大或缩小更新量。η scales the update amount.
RECOMPUTE
落点不同Different landing
下一步从不同位置重算梯度。Next gradients come from different positions.
TRAJECTORY
轨迹分叉Paths diverge
小差异累积成慢、稳或震荡。Small differences become slow, stable, or oscillating paths.
04最小机制:η 乘在梯度前Minimal Mechanism: η Multiplies the Gradient
Δθ = −η∇L
Δθ = −η∇L(θ)
η 太小:Δθ 很短;η 合适:持续下降;η
太大:一步跨过低谷,下一步梯度反向,再跨回来。Tiny η makes short updates; a suitable η keeps descending; excessive η crosses the
basin, flips the next gradient, and crosses back.
改变 ηChange η梯度方向不变direction unchanged
更新距离改变Update distance changes落在不同坡面land on a different slope
训练轨迹改变Training path changes慢、稳或发散slow, stable, or divergent
05边界实验:把学习率推到最大Boundary Test: Max Out the Learning Rate
主动制造失败Create a failure
让紫球反复跨谷Make the violet ball cross the basin
方向计算没有错,失败来自每一步太长。观察紫色轨迹在谷两侧跳动,而青球仍慢慢前进。The direction is correct; failure comes from steps that are too long. Watch violet
bounce while cyan still crawls.
尚未执行。Not run yet.
✗ 学习率越大,训练一定越快。A larger learning rate always trains faster.
→ ✓ 过大会震荡甚至发散。Too large can oscillate or diverge.
✗ 学习率改变下山方向。Learning rate changes the descent direction.
→ ✓ 它缩放梯度给出的更新。It scales the gradient update.
✗ 一个学习率适合所有曲面。One rate fits every surface.
→ ✓ 合适步长取决于地形与尺度。A suitable rate depends on geometry and scale.
06 · 一句话带走06 · One line to keep
学习率不决定往哪走,而决定沿负梯度一步走多远;太小慢,合适稳,太大越谷。Learning rate does not choose direction; it sets distance along the negative gradient:
too small is slow, suitable is stable, too large overshoots.