故事 · Tukey 与「诚实显著差异」
Origin Story · Tukey and the "Honest Significant Difference"
ANOVA 的 F 检验显著,只告诉你「这几组不全一样」,却不说是哪几组不同。
天真的做法是把每两组都做一次 t 检验 —— 但 4 组就有 6 次比较,每次冒 5% 的假阳性,合起来假阳性率飙到 26%:
你几乎注定会「发现」一个根本不存在的差异。1949 年,统计学家 John Tukey 给出解法:
用学生化极差分布(studentized range,符号 q)一次性校准整族比较的门槛,
让「族内至少一个假阳性」的概率被压回 α。他把这个统一门槛叫 HSD —— Honestly Significant Difference(诚实显著差异):
只有当两组均值差 |x̄ᵢ − x̄ⱼ| 超过 q·√(MSW/n) 时,才敢诚实地说「这两组真的不同」。
A significant ANOVA F only tells you "these groups are not all equal" — it does not say which pairs differ.
The naive fix is a t-test on every pair — but 4 groups mean 6 comparisons, each risking 5% false positives, so the combined rate rockets to 26%:
you are nearly guaranteed to "find" a difference that isn't there. In 1949, statistician John Tukey offered the cure:
use the studentized-range distribution (symbol q) to calibrate the threshold for the entire family of comparisons at once,
pushing the probability of "at least one false positive in the family" back down to α. He named this single threshold HSD — Honestly Significant Difference:
only when two group means differ by |x̄ᵢ − x̄ⱼ| more than q·√(MSW/n) may you honestly say "these two groups truly differ."
1 组均值 ± ½HSD · 两条不重叠 = 显著不同 Group Means ± ½HSD · Non-Overlapping = Significantly Different
HSD=—2 为什么不能裸做两两 t · 家族误差爆炸 Why Not Just Run Pairwise t · Family-Wise Error Explodes
红线:裸 t 的家族假阳性 1−(1−α)^m 随比较数 m 飙升;绿线:Tukey HSD 把它压平在 α=0.05。 Red: naive-t family-wise false positives 1−(1−α)^m climb with the number of comparisons m; green: Tukey HSD holds it flat at α=0.05.
3 动手算一遍 · 4 组显著后定位差异 Work It Out · Locating Differences After a Significant 4-Group ANOVA
背景:单因子 ANOVA 已显著。k=4 组,每组 n=5,MSW=6,组均值: Setup: one-way ANOVA already significant. k=4 groups, n=5 each, MSW=6, group means: x̄₁=20, x̄₂=23, x̄₃=27, x̄₄=30
dfW = N−k = 20−4 = 16 → q0.05, 4, 16 = 4.05 (查学生化极差表studentized-range table)
HSD = q·√(MSW/n) = 4.05·√(6/5) = 4.05·1.095 = 4.44
|x̄₁−x̄₂| = 3 < 4.44 → ns · |x̄₁−x̄₃| = 7 > 4.44 → sig
|x̄₁−x̄₄| = 10 > 4.44 → sig · |x̄₂−x̄₃| = 4 < 4.44 → ns
|x̄₂−x̄₄| = 7 > 4.44 → sig · |x̄₃−x̄₄| = 3 < 4.44 → ns
HSD = q·√(MSW/n) = 4.05·√(6/5) = 4.05·1.095 = 4.44
|x̄₁−x̄₂| = 3 < 4.44 → ns · |x̄₁−x̄₃| = 7 > 4.44 → sig
|x̄₁−x̄₄| = 10 > 4.44 → sig · |x̄₂−x̄₃| = 4 < 4.44 → ns
|x̄₂−x̄₄| = 7 > 4.44 → sig · |x̄₃−x̄₄| = 3 < 4.44 → ns
结论:6 对里 3 对显著(1-3、1-4、2-4)。相邻组两两拉不开,但跨度大的能分开 —— 组呈现「重叠梯队」:{1,2}、{2,3}、{3,4} 各自内部不可区分,但 1 明显低于 3、4,2 明显低于 4。若当初用 6 次裸 t,家族假阳性率已是 1−0.95⁶ = 26.5%。 Conclusion: of 6 pairs, 3 are significant (1-3, 1-4, 2-4). Adjacent groups don't separate, but wide spans do — the groups form "overlapping tiers": {1,2}, {2,3}, {3,4} are internally indistinguishable, yet 1 is clearly below 3 and 4, and 2 below 4. Had you run 6 naive t-tests, the family-wise false-positive rate would already be 1−0.95⁶ = 26.5%.
4 现实里的事后比较 Post-Hoc Comparisons in the Real World
选供应商/配方:5 家供应商良率 ANOVA 显著后,Tukey 告诉你到底哪几家真比别家好,而不是「至少有一家不同」这种空话。
Choosing a supplier/recipe: after 5 suppliers' yields test significant, Tukey tells you which ones are genuinely better — not the empty "at least one differs."
DOE 水平定位:某因子在实验里显著后,用多重比较找出「最优那一档」到底强在哪几个水平对上。
DOE level location: once a factor tests significant, multiple comparisons pinpoint exactly which level pairs drive the "best setting."
控制家族误差:比较越多、越要防「大海捞针捞出噪声」。Tukey 是等样本时的默认武器,Bonferroni/Scheffé 是其他场景的替补。
Control family error: the more comparisons, the more you must guard against "fishing noise out of the sea." Tukey is the default for equal n; Bonferroni/Scheffé cover other cases.
先总后分:永远先跑总的 ANOVA(F 检验)守门,再做事后比较;F 不显著就不该继续两两挖。
Omnibus first: always run the overall ANOVA (F test) as a gate, then do post-hoc; if F is not significant, don't go digging pairwise.
一句话In One Line
ANOVA 是总门卫,只回答「有没有组不一样」;Tukey HSD 是定位仪,回答「到底哪几组不一样」。
关键洞见是多重比较会累积假阳性:做 m 次独立比较,家族假阳性是 1−(1−α)^m,组一多就失控。
Tukey 用学生化极差 q(而非普通 t)算一个统一门槛 HSD = q·√(MSW/n),
把「族内至少一个假阳性」的概率钉回 α。两组均值差越过 HSD 才算真差异。
记住顺序:先 F 守门,再 Tukey 定位;F 不过就别两两挖 —— 那只是在噪声里钓鱼。
ANOVA is the gatekeeper, answering only "is any group different"; Tukey HSD is the locator, answering "which groups differ."
The key insight is that multiple comparisons accumulate false positives: run m independent tests and the family-wise rate is 1−(1−α)^m, which runs away as groups multiply.
Tukey uses the studentized range q (not the ordinary t) to build one shared threshold HSD = q·√(MSW/n),
pinning the probability of "at least one false positive in the family" back at α. A pair is a real difference only when its mean gap clears HSD.
Remember the order: F first to gate, then Tukey to locate; if F fails, don't dig pairwise — that is just fishing in noise.
常见误用Common Mistakes
跳过 ANOVA 直接两两 t。不校正的多重 t 会把假阳性从 5% 推到 20%+;先 F 守门,再用 Tukey 控制家族误差。
Skipping ANOVA and going straight to pairwise t. Uncorrected multiple t-tests push false positives from 5% to 20%+; gate with F first, then control family error with Tukey.
样本不等还硬用标准 Tukey。经典 HSD 假设各组等 n;样本不等时用 Tukey-Kramer 修正,否则门槛算错。
Using standard Tukey with unequal n. Classic HSD assumes equal n; with unequal group sizes use the Tukey-Kramer correction, or the threshold is wrong.
把「不显著」当成「相等」。相邻组没越过 HSD 只是「证据不足以区分」,不等于两组真的一样。
Reading "not significant" as "equal". Adjacent groups not clearing HSD only means "not enough evidence to separate them," not that they are truly equal.