S5-01 · 基础 · 概念实验 S5-01 · Foundation · Concept Lab

训练语料 · 模型吃过什么,会影响它怎么接话 Training Corpus · What a Model Reads Shapes How It Continues

corpus mix → repeated patterns → parameter pressure → completion bias

训练语料(Training Corpus)是用来学习模型参数的数据集合,其组成、质量和重复会塑造模型行为。相关偏向不等于模型逐字记住了每篇原文。 A training corpus is the data collection used to learn model parameters. Its composition, quality, and duplication shape behavior, but a learned tendency is not proof of verbatim memorization.

01 · 先从日常问题开始 01 · Start with an everyday question

一个人一个月只看菜谱,再让他补完“今天适合……”——你猜他会先想到什么? Someone reads only recipes for a month, then finishes “Today is good for…” What do you expect first?

02 调整专业语料占比,看续写偏向改变 Change Specialist Share and Watch Completion Bias Move

拖动语料桶占比 drag corpus mix
维修主题占比 Repair topic share
of 5 toy samples
重复率 Duplicate rate
exact duplicate share
维修续写偏向 Repair completion bias
toy probability

03 语料地图只需看四件事 A Corpus Map Needs Four Things

把刚才的变化拆开 Map the change to parts
01 · MIX

组成 Composition

不同主题各占多少,决定模型反复看见哪些模式。 Topic proportions determine which patterns the model sees repeatedly.

02 · QUALITY

质量 Quality

错误、过期和低质量样本也会参与参数更新。 Incorrect, stale, and low-quality samples still influence updates.

03 · REPEAT

重复 Duplication

同一文本重复出现,会让局部模式被过度强调。 Repeated text can overemphasize local patterns.

04 · BOUNDARY

证据边界 Evidence boundary

观察到偏向只说明相关,不能据此断言记住某篇原文。 Observed bias shows association; it does not prove memorization of a specific source.

04 玩具机制:主题曝光推动续写偏向 Toy Mechanism: Topic Exposure Pushes Completion Bias

bias ≈ 20% + 0.7 × share
biasrepair ≈ 20% + 0.7 × corpus share
这是 5 样本概念地图,不是真实训练。固定种子只负责稳定展示顺序;比例变化用于演示因果方向。 This is a five-sample concept map, not real training. The fixed seed stabilizes display order; the ratio illustrates causal direction only.
增减语料桶 Change bucket mix 维修文本曝光变化 repair exposure changes
重复模式加权 Weight repeated patterns 参数更新压力改变 update pressure shifts
续写偏向移动 Completion bias moves 不是原文记忆证明 not proof of memorization

05 边界实验:把重复当成更多知识 Boundary Test: Mistake Duplicates for More Knowledge

主动制造失败 Create a failure

同一维修样本复制三次 Copy the same repair sample three times

打开重复模式并把维修占比拉到 80%。主题偏向会变强,但语料多样性反而下降。 Enable duplicates and raise repair share to 80%. Topic bias grows while corpus diversity falls.

尚未执行。先在上方改变主变量。 Not run yet. Change the main variable above first.
语料越多,质量一定越高。 More corpus always means better quality. → ✓ 组成、质量与重复必须分开检查。 Inspect composition, quality, and duplication separately.
模型偏向维修,说明记住了某篇文章。 A repair bias proves the model memorized one article. → ✓ 行为相关不等于逐字记忆证据。 Behavioral correlation is not proof of verbatim memory.
复制同一文本能增加知识多样性。 Copying one text increases knowledge diversity. → ✓ 重复只增加曝光,不增加独立信息。 Duplication increases exposure, not independent information.
06 · 一句话带走 06 · One line to keep

训练语料的组成、质量和重复会塑造模型行为;调语料桶能看到偏向变化,但“相关”不能被说成“记住了原文”。 Corpus composition, quality, and duplication shape model behavior. Changing the bucket reveals bias, but association must not be claimed as memorization.