Pretraining · AI.WIKI

01 · 先从日常问题开始 01 · Start with an everyday question

刚拆封的收音机只有随机旋钮位置；要让它稳定收到台，是改一句说明书，还是反复校准内部旋钮？ A new radio starts with random knob positions. Does one instruction tune it, or do repeated adjustments change its internals?

02 拖动训练时间轴，冻结任一步比较续写 Scrub the Training Timeline and Freeze Any Step

拖动批次步数 drag batch step

唯一主变量 One main variable

时间轴只有 0–4 个玩具批次。每前进一步，参数状态与同一提示下的续写分布一起变化。 The timeline has only four toy batches. Each step changes parameter state and completion distribution for the same prompt.

已完成批次 Completed batches 0/4

固定种子 Fixed seed 20260722

教学模拟值，非真实模型输出 simulated for teaching — not real model output

参数更新步 Parameter updates

—

toy batches

通用续写概率 General completion

—

toy probability

已保存检查点 Saved checkpoints

—

recoverable snapshots

03 预训练概念地图的四个层次 Four Layers in the Pretraining Concept Map

把刚才的变化拆开 Map the change to parts

01 · START

随机参数 Random parameters

起点只是一组初始数值，尚未稳定表达语料中的规律。 The start is merely initial numbers, before stable corpus patterns are learned.

02 · DATA

大规模语料 Large corpus

真实预训练规模巨大；本页只保留 5 样本玩具地图。 Real pretraining is enormous; this page keeps only a five-sample toy map.

03 · OBJECTIVE

通用目标 General objective

用统一学习目标反复计算差异并推动参数更新。 A general learning objective repeatedly measures error and drives updates.

04 · SNAPSHOT

检查点 Checkpoint

途中保存版本，以便恢复、评估和比较，不等于训练结束。 Intermediate snapshots support recovery and evaluation; they do not necessarily mean training is finished.

04 玩具机制：批次更新逐步改变分布 Toy Mechanism: Batch Updates Gradually Move a Distribution

θₜ₊₁ = θₜ − ηgₜ

θ_t+1 = θ_t − ηg_t

θ 表示参数，g 表示当前批次推动的方向。这里只画概念变化，不运行真实模型或真实梯度训练。 θ denotes parameters and g the direction from the current batch. This is a conceptual animation, not real model training or real gradients.

读取一个批次 Read one batch 固定玩具样本 fixed toy samples

计算更新方向 Compute update direction 通用目标给出差异 objective supplies error

保存阶段快照 Save stage snapshot 同一提示可回放比较 replay the same prompt

05 边界实验：把随机起点当成预训练模型 Boundary Test: Treat Random Initialization as Pretrained

主动制造失败 Create a failure

停在第 0 批次却要求稳定续写 Stop at batch zero and expect stable completion

把时间轴拉回 0。参数没有从语料获得任何更新，同一提示只显示固定种子的随机起点。 Return the timeline to zero. Parameters have received no corpus updates; the prompt shows only a fixed-seed random starting point.

尚未执行。先在上方改变主变量。 Not run yet. Change the main variable above first.

✗ 预训练就是写一个更长的提示。 Pretraining means writing a longer prompt. → ✓ 预训练在训练阶段改变模型参数。 Pretraining changes model parameters during training.

✗ 检查点就是最终最佳模型。 A checkpoint is automatically the final best model. → ✓ 检查点只是可恢复、可评估的阶段快照。 A checkpoint is a recoverable, evaluable stage snapshot.

✗ 这条玩具曲线就是真实训练结果。 This toy curve is a real training result. → ✓ 本页只展示概念方向，不做真实训练。 This page shows conceptual direction, not real training.

06 · 一句话带走 06 · One line to keep

预训练用通用目标在大规模语料上反复更新初始参数，并沿途保存检查点；它改变参数，不是提示或检索的别名。 Pretraining repeatedly updates initial parameters on large-scale data under a general objective and saves checkpoints along the way. It changes parameters; it is not prompting or retrieval.