KV Cache · Trade Memory for Less Repeated Compute

01 · 做长账时留着中间结果 01 · Keeping intermediate results in a long calculation

为什么记住前面算过的表格能让下一步更快，却会让桌面越堆越满？ Why does keeping earlier results make the next step faster while steadily filling the desk?

02 逐 token 生成，开关缓存看算力与显存交换 Generate Token by Token and Toggle the Cache

速度 ↔ 显存 Speed ↔ memory

状态 A Metric A

—

状态 B Metric B

—

状态 C Metric C

—

直接操作主变量，再拖回去验证一句话结论。所有数值均为固定种子的教学模拟。 Operate the main variable, then move it back to verify the one-line conclusion. All values are fixed-seed teaching simulations.

03 把变化拆成 4 个零件 Break the Change into Four Parts

KV Cache

PART 01

过去 K Past Keys

每个旧 token 的匹配表示可直接从缓存读取。 The matching representation of each past token can be read from cache.

PART 02

过去 V Past Values

每个旧 token 的内容表示同样保留，下一步不必重算。 Each past token's content representation is retained and need not be recomputed.

PART 03

新 token New token

每一步只为新位置计算一组 K/V，再把它追加进仓库。 Each step computes one new K/V pair and appends it to the warehouse.

PART 04

显存仓 Memory warehouse

序列越长，缓存格越多；速度收益以显存占用为代价。 Longer sequences consume more slots; the speedup costs memory.

04 最小机制：旧 K/V 复用，新 K/V 追加 Minimal Mechanism: Reuse Old K/V and Append New K/V

KV Cache

cost_{no cache} ∝ 1+…+n · cost_cache ∝ n

教学格数展示重复前缀计算的差异；实际成本还受批次、实现和硬件影响。 Teaching cells show the repeated-prefix difference. Real cost also depends on batching, implementation, and hardware.

生成新 token Generate new token 需要读取过去上下文 past context is needed

命中缓存 Hit KV cache 旧前缀不再重算 old prefix is not recomputed

追加 K/V Append K/V 速度提高，显存增长 speed rises, memory grows

05 边界实验：把显存仓塞到溢出 Boundary Test: Fill the Memory Warehouse Past Capacity

主动制造失败 Create a failure

缓存不是无限记忆 The cache is not unlimited memory

让序列增长到 8 个 token。重复计算区仍消失，但 16 个 K/V 格超过 12 格教学容量，出现溢出。 Grow the sequence to eight tokens. Repeated compute still disappears, but 16 K/V slots exceed the 12-slot teaching capacity.

尚未执行。先在主交互中观察正常机制。 Not run yet. Observe the normal mechanism in the main interaction first.

✗ KV Cache 会让模型记住跨会话事实。 KV Cache gives the model cross-session memory. → ✓ 它缓存当前生成中的中间 K/V，不是长期记忆。 It caches intermediate K/V for the current generation, not long-term memory.

✗ 打开缓存就不再做注意力计算。 Cache removes all attention computation. → ✓ 新 token 仍要读取旧 K/V 并计算自己的 K/V。 The new token still reads past K/V and computes its own pair.

✗ 缓存越大只有好处。 A larger cache is only beneficial. → ✓ 上下文增长会持续占用显存并影响吞吐。 Growing context continuously consumes memory and affects throughput.

06 · 一句话带走 06 · One line to keep

KV Cache 复用过去 token 的 K/V，让重复计算区消失、生成更快；逐 token 点击也能看到显存仓同步占满。 A KV Cache reuses past K/V, removing repeated compute and speeding generation. Step token by token to see the memory warehouse fill at the same time.