Leave-one-out error

For broader coverage of this topic, see Cross-validation (statistics).

For mathematical analysis and statistics, Leave-one-out error can refer to the following:

• Leave-one-out cross-validation Stability (CVloo, for stability of Cross Validation with leave one out): An algorithm f has CVloo stability β with respect to the loss function V if the following holds:

Illustration of leave-one-out cross-validation (LOOCV) when n = 8 observations. A total of 8 models will be trained and tested.

$\forall i\in \{1,...,m\},\mathbb {P} _{S}\{\sup _{z\in Z}|V(f_{S},z_{i})-V(f_{S^{|i}},z_{i})|\leq \beta _{CV}\}\geq 1-\delta _{CV}$

• Expected-to-leave-one-out error Stability ($Eloo_{err}$, for Expected error from leaving one out): An algorithm f has $Eloo_{err}$ stability if for each n there exists a$\beta _{EL}^{m}$ and a $\delta _{EL}^{m}$ such that:

$\forall i\in \{1,...,m\},\mathbb {P} _{S}\{|I[f_{S}]-{\frac {1}{m}}\sum _{i=1}^{m}V(f_{S^{|i}},z_{i})|\leq \beta _{EL}^{m}\}\geq 1-\delta _{EL}^{m}$, with $\beta _{EL}^{m}$and $\delta _{EL}^{m}$ going to zero for $n\rightarrow \inf$

Preliminary notations

With X and Y being a subset of the real numbers R, or X and Y ⊂ R, being respectively an input space X and an output space Y, we consider a training set:

$S=\{z_{1}=(x_{1},\ y_{1})\ ,..,\ z_{m}=(x_{m},\ y_{m})\}$ of size m in $Z=X\times Y$ drawn independently and identically distributed (i.i.d.) from an unknown distribution, here called "D". Then a learning algorithm is a function $f$ from $Z_{m}$ into $F\subset YX$ which maps a learning set S onto a function $f_{S}$ from the input space X to the output space Y. To avoid complex notation, we consider only deterministic algorithms. It is also assumed that the algorithm $f$ is symmetric with respect to S, i.e. it does not depend on the order of the elements in the training set. Furthermore, we assume that all functions are measurable and all sets are countable which does not limit the interest of the results presented here.

The loss of an hypothesis f with respect to an example $z=(x,y)$ is then defined as $V(f,z)=V(f(x),y)$. The empirical error of f can then be written as $I_{S}[f]={\frac {1}{n}}\sum V(f,z_{i})$.

The true error of f is $I[f]=\mathbb {E} _{z}V(f,z)$

Given a training set S of size m, we will build, for all i = 1....,m, modified training sets as follows:

• By removing the i-th element

$S^{|i}=\{z_{1},...,\ z_{i-1},\ z_{i+1},...,\ z_{m}\}$

• and/or by replacing the i-th element

$S^{i}=\{z_{1},...,\ z_{i-1},\ z_{i}',\ z_{i+1},...,\ z_{m}\}$

See also

• Constructive analysis
• History of calculus
• Hypercomplex analysis
• Jackknife resampling
• Statistical classification
• Timeline of calculus and mathematical analysis

References

• S. Mukherjee, P. Niyogi, T. Poggio, and R. M. Rifkin. Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Adv. Comput. Math., 25(1-3):161–193, 2006