Hypothesis Testing Decision Errors
Hypothesis Testing sometimes mistake - and we need to have tools quantify these mistakes
Type I and Type II Errors
Summary [1]
| |
$H_0$ is true |
$H_0$ is false
|
| Reject $H_0$
|
Type I error
False positive
|
Correct outcome
True positive
|
| Fail to reject $H_0$
|
Correct outcome
True negative
|
Type II error
False negative
|
- A decrease in one type of error leads to increase the probability of other
- So we need to have more evidence
Type I Error
- Reject $H_0$ when it's true
- This happens with probability \alpha
- (An innocent is falsely convicted)
Significance Level $\alpha$ controls Type I errors
Controlling Family-Wise Error Rate
- suppose we run Multiple Comparisons Tests
- e.g. want to compare pair-wise 10 samples
- thus we need to make about $\sum_{i=1}^{10} i = 45$ comparisons
- the chances hight that among the 45 tests a couple of them will incorrectly reject $H_0$ - i.e. they will make Type 1 Error
- the solution is to modify the significance level, e.g. using the Bonferroni Correction
- see Family-Wise Error Rate
Type II Error
- Fail to reject $H_0$ when $H_A$ is true
- This happens with probability $\beta = 1 - \text{power}$
- We don't have enough power - probably the test size is too small
- (A criminal is freed)
The probability of making Type II Errors is called the Type II error rate
Controlling Type II Errors
Type II Errors can be controlled by:
Sample Size
Power of a test also allows to control the
- suppose the power of a test is 0.979. what's the type II error rate?
- it's 1 - 0.979 = 0.021 - this is the probability of failing to reject $H_0$ when it's true
Sources