Hypothesis Testing Decision Errors
Hypothesis Testing sometimes mistake  and we need to have tools quantify these mistakes
Type I and Type II Errors
Summary [1]

$H_0$ is true 
$H_0$ is false

Reject $H_0$

Type I error False positive

Correct outcome True positive

Fail to reject $H_0$

Correct outcome True negative

Type II error False negative

 A decrease in one type of error leads to increase the probability of other
 So we need to have more evidence
Type I Error
 Reject $H_0$ when it's true
 This happens with probability \alpha
 (An innocent is falsely convicted)
Significance Level $\alpha$ controls Type I errors
Controlling FamilyWise Error Rate
 suppose we run Multiple Comparisons Tests
 e.g. want to compare pairwise 10 samples
 thus we need to make about $\sum_{i=1}^{10} i = 45$ comparisons
 the chances hight that among the 45 tests a couple of them will incorrectly reject $H_0$  i.e. they will make Type 1 Error
 the solution is to modify the significance level, e.g. using the Bonferroni Correction
 see FamilyWise Error Rate
Type II Error
 Fail to reject $H_0$ when $H_A$ is true
 This happens with probability $\beta = 1  \text{power}$
 We don't have enough power  probably the test size is too small
 (A criminal is freed)
The probability of making Type II Errors is called the Type II error rate
Controlling Type II Errors
Type II Errors can be controlled by:
Sample Size
Power of a test also allows to control the
 suppose the power of a test is 0.979. what's the type II error rate?
 it's 1  0.979 = 0.021  this is the probability of failing to reject $H_0$ when it's true
Sources