Latency and Throughput

01 · 收银台排队变长，为什么每分钟结账数却不再增加01 · Why does the queue grow while checkouts per minute stop rising?

并发从 8 拉到 32 后，每秒任务数只多了 3%，p95 却翻倍。继续加请求是在扩容，还是只把等待藏进队列？Concurrency rises from 8 to 32. Tasks per second gain only 3%, while p95 latency doubles. Is that more capacity or merely more waiting hidden in a queue?

02拖动并发与批量，让 p50/p95、每秒任务数与饱和队列同步变化Drag Concurrency and Batch Size to Move p50, p95, Tasks per Second, and the Saturation Queue Together

容量曲线 · 饱和拐点capacity curve · saturation knee

状态 AMetric A

—

状态 BMetric B

—

状态 CMetric C

—

所有概率、分数、成本、token 与输出均为固定种子的教学模拟，不代表真实模型输出。把主变量拖回即可复核结论。All probabilities, scores, costs, tokens, and outputs are fixed-seed teaching simulations, not real model output. Move the main variable back to verify the conclusion.

03把刚才的变化拆成 4 个零件Break the Change into Four Parts

Latency and Throughput

LAT

单次延迟Per-request latency

p50 看典型体验，p95 看尾部等待。p50 shows typical experience; p95 exposes tail waiting.

TPS

吞吐Throughput

tasks/s 衡量单位时间完成多少工作。Tasks/s measures completed work per second.

BATCH

批处理Batching

适量批处理提高设备利用率，也会增加凑批等待。Moderate batching raises utilization but adds batch-assembly wait.

KNEE

饱和点Saturation point

到达容量上限后吞吐趋平，队列与尾延迟陡升。After capacity is reached, throughput flattens while queues and tail latency surge.

04最小机制：主交互能逐项验证Minimal Mechanism: Verify Each Link in the Interaction

MECHANISM

queue ≈ max(0, offered_load − service_capacity)

主交互用并发与批量估计 offered_load 和 capacity；超过饱和点后，新增负载主要进入队列而不是完成量。The interaction estimates offered load and capacity from concurrency and batch size. Beyond saturation, added load becomes queue depth rather than completed work.

提高负载Raise load并发 × 批量concurrency × batch

接近容量Approach capacity吞吐增长throughput rises

越过饱和Cross saturationp95 + 队列陡升p95 + queue surge

05把并发和批量同时推到最大Push Concurrency and Batch Size to Maximum

主动制造失败Create a failure

边界实验Boundary experiment

若只看吞吐，你会错过尾延迟和排队失控。实验应显示饱和点后 tasks/s 趋平、p95 与队列骤增。Throughput alone hides runaway tail latency and queues. Beyond the knee, tasks/s must flatten while p95 and queue depth surge.

尚未执行。先在主交互中观察正常机制。Not run yet. Observe the normal mechanism in the main interaction first.

✗ 吞吐越高，单次响应一定越快。Higher throughput always means faster responses.→ ✓ 接近饱和时吞吐可高而 p95 已恶化。Near saturation, throughput can stay high while p95 deteriorates.

✗ 只报平均延迟足够。Average latency is enough.→ ✓ 至少同时看 p50 与 p95，避免尾部被平均。Track at least p50 and p95 so the tail is not averaged away.

✗ 并发翻倍就能让吞吐翻倍。Doubling concurrency doubles throughput.→ ✓ 容量上限后只会积累排队与超时。Beyond capacity it only accumulates queues and timeouts.

06 · 一句话带走06 · One line to keep

延迟描述一次任务等多久，吞吐描述每秒完成多少；并发和批量越过饱和点后，吞吐趋平而 p95 与队列会骤增。Latency tells how long one task waits; throughput tells how many finish per second. Past saturation, throughput flattens while p95 and queue depth surge.