# ML Wiki

## Hypothesis Testing Decision Errors

Hypothesis Testing sometimes mistake - and we need to have tools quantify these mistakes

## Type I and Type II Errors

Summary [1]
$H_0$ is true $H_0$ is false
Reject $H_0$ Type I error
False positive
Correct outcome
True positive
Fail to reject $H_0$ Correct outcome
True negative
Type II error
False negative

• A decrease in one type of error leads to increase the probability of other
• So we need to have more evidence

## Type I Error

• Reject $H_0$ when it's true
• This happens with probability \alpha
• (An innocent is falsely convicted)

Significance Level $\alpha$ controls Type I errors

### Controlling Family-Wise Error Rate

• suppose we run Multiple Comparisons Tests
• e.g. want to compare pair-wise 10 samples
• thus we need to make about $\sum_{i=1}^{10} i = 45$ comparisons
• the chances hight that among the 45 tests a couple of them will incorrectly reject $H_0$ - i.e. they will make Type 1 Error
• the solution is to modify the significance level, e.g. using the Bonferroni Correction
• see Family-Wise Error Rate

## Type II Error

• Fail to reject $H_0$ when $H_A$ is true
• This happens with probability $\beta = 1 - \text{power}$
• We don't have enough power - probably the test size is too small
• (A criminal is freed)

The probability of making Type II Errors is called the Type II error rate

### Controlling Type II Errors

Type II Errors can be controlled by:

### Power of a Test

Power of a test also allows to control the

• suppose the power of a test is 0.979. what's the type II error rate?
• it's 1 - 0.979 = 0.021 - this is the probability of failing to reject $H_0$ when it's true