The cheat sheet

Use this as a reference after working through an example. For a question that feels unreadable, start with a notation walkthrough.

Identify the objects

symbol	meaning	example
$X$, $x$	random input; a particular input	a campaign; its measured spend
$X_j$, $x_j$	feature j	TV spend within one input
$(x_i,y_i)$	training row i	that campaign’s spend and sales
p, N	number of features; number of examples	3 features, 200 campaigns
$f(x)$	true conditional mean	$\mathbb E[Y\mid X=x]$
$\hat f_D(x)$	prediction from a model fitted on D	changes if you train on different data
$\varepsilon$	output minus conditional mean	$Y-f(X)$
$\sigma^2(x)$	conditional output variance	constant $\sigma^2$ in our simulations
$\theta$, $\hat\theta$	candidate parameters; fitted parameters	intercept and slopes
$\beta^\top x$	dot product	$\sum_{j=1}^{p}\beta_jx_j$
$\operatorname{argmin}_\theta$	parameter choice with the lowest score	returns parameters, not the minimum score

What is the expectation over?

expression	hold fixed	average over
$\mathbb E[Y\mid X=x]$	input x	possible outputs at x
$\mathbb E_D[\hat f_D(x)]$	input and training method	training datasets D
$\mathbb E[(Y-g(X))^2]$	prediction rule g	new input–output pairs
$\frac1M\sum_i(y_i-\hat f(x_i))^2$	fitted model and held-out dataset	the finite list of observed errors

Definitions and assumptions

Define $f(x)=\mathbb E[Y\mid X=x]$ and $\varepsilon=Y-f(X)$. Then $\mathbb E[\varepsilon\mid X=x]=0$ by definition (when the conditional mean exists).

Constant conditional variance, $\operatorname{Var}(\varepsilon\mid X=x)=\sigma^2$, is an extra assumption. Independent noise is stronger: fixing X leaves its entire distribution unchanged. The examples use independent, constant-variance noise; the conditional-mean result does not require those simplifications.

Squared loss: the three related results

For one prediction c, assuming finite second moments:

$\mathbb E[(Y-c)^2]=\operatorname{Var}(Y)+(\mathbb E[Y]-c)^2.$

The best c is the mean. Absolute loss instead selects a median.

For a fixed predictor h at a fixed input x:

$\mathbb E[(Y-h(x))^2\mid X=x]=(f(x)-h(x))^2+\sigma^2(x).$

For a training method repeated across datasets, scored on an independent fresh output at x:

$$\begin{aligned} R(x)&=\operatorname{Bias}(x)^2+\operatorname{Var}_D(\hat f_D(x))+\sigma^2(x),\\ \operatorname{Bias}(x)&=\mathbb E_D[\hat f_D(x)]-f(x). \end{aligned}$$

For overall expected error, average $R(x)$ over inputs too. A finite test-set MSE is an estimate of one fitted model’s performance; it is not exactly this population decomposition.

Choosing and assessing a model

1. Fit parameter values on training data.
2. Use validation data or cross-validation to choose degree, regularization, or model family.
3. Assess the frozen choice on separate test data.

Minimum unpenalized training error cannot increase when the candidate families are nested and the same objective is minimized on the same data. A U-shaped expected-error curve is a classical example, not a universal law.

More data often stabilizes fitted predictions. It does not let a straight line represent arbitrary curvature. The bias of an estimator can also depend on sample size.

Neighbors in p dimensions

For uniform points in a unit-radius ball and a radius-r ball at its center, with 0≤r≤1:

$\Pr(\text{inside})=r^p,\qquad \mathbb E[\text{count}]=Nr^p.$

To get k expected neighbors, $r=(k/N)^{1/p}$. For N=1,000 and k=10, r is about 0.63 in ten dimensions. That is 63% of the radius but 1% of the volume. Boundary effects, nonuniform sampling, and lower-dimensional structure change this calculation.

For a unit ball inside a cube of side 2, the fraction of cube volume is

$\frac{\pi^{p/2}}{2^p\Gamma(p/2+1)}.$

Frequent distinctions

Output variance versus prediction variance

Output variance describes the possible Y values at one x. Prediction variance describes the predictions from repeatedly trained models at that same x. More training examples can reduce the second while the first stays unchanged.

Can a measured error be below the noise floor?

Yes. The floor bounds expected squared error using the specified inputs. A finite sample can have unusually small noise; training also adapts the model to its particular sample. A low training score alone does not diagnose overfitting.

Is irreducible error permanently irreducible?

It is relative to the measured inputs. A useful additional feature can explain variation that was previously noise. That may reduce conditional output variance, while also making estimation more demanding with a finite sample.

Practice without topic labels →