# ML Wiki

## True Error of Model

What do we do when we want to know how accurately the model will perform in practice

• need to estimate its true error

### Estimating the Accuracy

Given:

• classification model $C$
• dataset $S$ with $n$ examples drawn w.r.t. distribution $P$

Problem

• estimate the accuracy of $C$ over future instances drawn with $P$
• this is called the true error
• it's important to distinguish sample error and true error

### Sample Error

the sample error of $C$ calculated on sample $S$ is

• the proportion of examples in $S$ that $C$ misclassified
• $\text{error}(C, S) = \cfrac{1}{| S |} \sum_{(x,y) \in S} \delta (C(x) \ne y)$
• $\text{acc}(C, S) = \cfrac{1}{| S |} \sum_{(x,y) \in S} \delta (C(x) = y)$

But usually we have training and testing sets (see Cross-Validation)

• i.e. we have some data set $S$ (drawn from the population with distribution $P$)
• learning set $R \subset S$,
• training set $T \subset S$,
• $R$ and $T$ are disjoint: $R \cap T = \varnothing$
• so the sample error is computed against $T$: $\text{error}(C, T)$

### True Error

the true error of $C$ w.r.t distribution $S$ on the population $D$

• is the probability to misclassify an instance drawn from $D$ at random
• $\text{error}(C, D) = \sum_{(x,y) \in D} P(x, y) \cdot \delta(C(x) \ne y)$
• $P(x, y)$ is the probability to draw a pair $(x,y) \in D$

Estimate of $\text{error}(C, D)$

• Sample error $\text{error}(C, S)$ is just an estimate of $\text{error}(C, D)$
• since $S \subset D$, $S$ is always finite, while $D$ can be infinite
• but this estimate is not always accurate! (need to have more accurate estimate)

More accurate estimates: