ML Wiki

Family-Wise Error Rate

Controlling FWER

• suppose we run many tests at the same time
• we perform a Pairwise t-test and want to compare 10 samples
• thus we need to make about $\sum_{i=1}^{10} i = 45$ comparisons
• the chances hight that among the 45 tests a couple of them will incorrectly reject $H_0$ - i.e. they will make Type 1 Error about 5% of the time at $\alpha = 0.05%$
• the solution: to modify the significance level - run the tests at significance level $\alpha^* < \alpha$

Corrections

Bonferroni Correction

use $\alpha^* = \alpha \cdot \cfrac{1}{K}$

• where $K$ is the number of tests to run
• in a pairwise test of $k$ samples, $K= \cfrac{k \cdot (k - 1)}{2}$

Sources

Machine Learning Bookcamp: Learn machine learning by doing projects. Get 40% off with code "grigorevpc".