S5-20 · 高阶 · 3DS5-20 · Expert · 3D

缩放定律 · 参数、数据、算力必须一起配平Scaling Laws · Balance Parameters, Data, and Compute

parameters × data × compute budget → empirical loss surface → balanced region

缩放定律(Scaling Laws)描述模型规模、数据量、计算量与损失之间的经验关系。它提醒我们:只加参数而不配数据和算力,配置会失衡。Scaling laws describe empirical relationships among model size, data, compute, and loss. Increasing parameters without matching data and compute produces an imbalanced configuration.

01 · 厨房扩十倍,食材与火力不变01 · Ten times the kitchen, same ingredients and heat

只把参数规模拉高,却不给更多数据与算力,这台更大的模型为什么可能只是“更大的空厨房”?If parameters grow while data and compute stay fixed, why can a larger model become little more than a bigger empty kitchen?

02在 3D 参数—数据—算力预算面上移动配置,失衡点自动标红Move a Configuration on the 3D Parameter–Data–Compute Budget Surface

3D 预算面 · 三轴配平3D budget surface · three-way balance
总预算指数Budget index
173
P + D + C
预测损失Predicted loss
1.12
toy L(P,D,C)
配置状态Configuration
BALANCED
three-way allocation
正在装载本地 3D 场景…若持续显示,需要 WebGL / three.js 未加载。Loading the local 3D scene… If this remains visible, WebGL is required or three.js did not load.
P 55 · D 60 · C 58
CONFIG
P
D
C
拖拽旋转 · 滚轮缩放 · 停手后缓转drag to orbit · wheel to zoom · idle auto-rotate

03把刚才的变化拆成 4 个零件Break the Change into Four Parts

Scaling Laws
PARAMETERS

参数规模Parameters

模型容量轴;单独增大可能吃不满。The capacity axis; increasing it alone can leave capacity underused.

DATA

数据量Data

训练信号轴;不足会让大模型反复看少量样本。The signal axis; too little data makes a large model recycle few examples.

COMPUTE

计算预算Compute

把参数与数据真正用于更新的训练工作量。Training work that actually combines parameters and data.

BALANCE

经验配平Empirical balance

同预算面上,三者配比决定预测损失。On one budget surface, allocation among all three shapes predicted loss.

04最小机制:主交互能逐项验证Minimal Mechanism: Verify Each Link in the Interaction

MECHANISM
L(P,D,C) ≈ L + aP−α + bD−β + compute_gap
三轴滑块共同定位一个配置点;与等预算面上的配平点距离越大,玩具预测损失越高并标红。The three sliders locate one configuration. Greater distance from the balanced region on the equal-budget surface raises toy predicted loss and turns the marker red.
设总预算Set budget形成等预算面form an equal-budget surface
分配 P / D / CAllocate P / D / C配置点落在面上或偏离place or miss the surface
比较经验损失Compare loss配平优于单轴堆叠balance beats one-axis scaling

05只把参数拉满,数据与算力保持低位Max Out Parameters While Data and Compute Stay Low

主动制造失败Create a failure

边界实验Boundary experiment

红点看起来“模型更大”,但它远离配平区;玩具损失反而上升,说明缩放定律不是只看参数。The red point is a larger model, yet far from the balanced region. Toy loss rises, showing that scaling laws are not parameter-only rules.

尚未执行。先在主交互中观察正常机制。Not run yet. Observe the normal mechanism in the main interaction first.
参数越多,损失必然越低。More parameters always lower loss.→ ✓ 数据与算力不足会让容量无法兑现。Insufficient data and compute can leave capacity unrealized.
缩放定律是每个模型都精确成立的物理定律。Scaling laws are exact physical laws for every model.→ ✓ 它们是特定范围与设置下的经验关系。They are empirical relationships for particular regimes and setups.
预算面上的最优点永远不变。The optimum on a budget surface never moves.→ ✓ 架构、数据质量与硬件会改变经验系数。Architecture, data quality, and hardware change the empirical coefficients.
06 · 一句话带走06 · One line to keep

缩放定律看参数、数据与算力的共同经验关系;在同一预算下,配平三轴通常比只堆一个规模数字更有意义。Scaling laws relate parameters, data, and compute empirically. Under one budget, balancing the three is usually more meaningful than maximizing a single scale number.