Hypothesis Testing Decision Errors

Hypothesis Testing sometimes mistake - and we need to have tools quantify these mistakes

Type I and Type II Errors

Summary [1]
$H_0$ is true $H_0$ is false
Reject $H_0$ Type I error
False positive
Correct outcome
True positive
Fail to reject $H_0$ Correct outcome
True negative
Type II error
False negative

  • A decrease in one type of error leads to increase the probability of other
  • So we need to have more evidence

Type I Error

  • Reject $H_0$ when it's true
  • This happens with probability \alpha
  • (An innocent is falsely convicted)

Significance Level $\alpha$ controls Type I errors

Controlling Family-Wise Error Rate

  • suppose we run Multiple Comparisons Tests
  • e.g. want to compare pair-wise 10 samples
  • thus we need to make about $\sum_{i=1}^{10} i = 45$ comparisons
  • the chances hight that among the 45 tests a couple of them will incorrectly reject $H_0$ - i.e. they will make Type 1 Error
  • the solution is to modify the significance level, e.g. using the Bonferroni Correction
  • see Family-Wise Error Rate

Type II Error

  • Fail to reject $H_0$ when $H_A$ is true
  • This happens with probability $\beta = 1 - \text{power}$
  • We don't have enough power - probably the test size is too small
  • (A criminal is freed)

The probability of making Type II Errors is called the Type II error rate

Controlling Type II Errors

Type II Errors can be controlled by:

Sample Size

Power of a Test

Power of a test also allows to control the

  • suppose the power of a test is 0.979. what's the type II error rate?
  • it's 1 - 0.979 = 0.021 - this is the probability of failing to reject $H_0$ when it's true