Multiple Comparisons Tests

Family-Wise Error Rate

Controlling FWER

  • suppose we run many tests at the same time
  • we perform a Pairwise t-test and want to compare 10 samples
  • thus we need to make about $\sum_{i=1}^{10} i = 45$ comparisons
  • the chances hight that among the 45 tests a couple of them will incorrectly reject $H_0$ - i.e. they will make Type 1 Error about 5% of the time at $\alpha = 0.05%$
  • the solution: to modify the significance level - run the tests at significance level $\alpha^* < \alpha$


Bonferroni Correction

use $\alpha^* = \alpha \cdot \cfrac{1}{K}$

  • where $K$ is the number of tests to run
  • in a pairwise test of $k$ samples, $K= \cfrac{k \cdot (k - 1)}{2}$


Machine Learning Bookcamp: Learn machine learning by doing projects. Get 40% off with code "grigorevpc".

Share your opinion