The best guess is the average

You must predict one number before seeing an output. To decide what counts as a good prediction, first decide how to score a mistake.

Score a guess

Let $c$ be your guess and $Y$ the output. A loss function assigns a cost to the error. Squared loss charges $(Y-c)^2$. Its expected cost is

$$\mathbb E[(Y-c)^2].$$

Hold $c$ fixed. For each possible $Y$, subtract $c$, square the difference, and average using the probabilities of the outputs. The expectation averages over outputs, not over guesses.

First, calculate with three numbers

Suppose $Y$ is equally likely to be 1, 3, or 5. Its mean is 3. Compare guessing 2 with guessing 3:

output	squared error if you guess 2	squared error if you guess 3
1	1	4
3	1	0
5	9	4
average	$11/3$	$8/3$

Guessing the mean costs 1 less. That is the squared distance between the two guesses: $(3-2)^2=1$. The same relationship holds for every guess. Let’s derive it.

The left graph shows individual errors. The right graph shows their average squared cost for different guesses.

Why the mean wins

Write $m=\mathbb E[Y]$. We want to separate variation in the output from the effect of choosing the wrong guess. Adding and subtracting $m$ lets us make that split without changing the error:

$$Y-c=\underbrace{(Y-m)}_{\text{output minus mean}}+\underbrace{(m-c)}_{\text{mean minus guess}}.$$

In the example, guessing 2 gives $Y-2=(Y-3)+1$. Now use $(a+b)^2=a^2+2ab+b^2$:

$$\begin{aligned} (Y-c)^2&=(Y-m)^2+2(Y-m)(m-c)\\ &\quad +(m-c)^2. \end{aligned}$$

Average each term. This is linearity of expectation.

term	its average	reason
$(Y-m)^2$	$\operatorname{Var}(Y)$	definition of variance
$2(Y-m)(m-c)$	$0$	$m-c$ is fixed; $\mathbb E[Y-m]=0$
$(m-c)^2$	$(m-c)^2$	it is already a fixed number

For the three-number example, the middle term takes values −4, 0, and 4: they cancel. The other terms average to $8/3$ and 1.

$$\mathbb E[(Y-c)^2]=\operatorname{Var}(Y)+(m-c)^2.$$

The variance does not depend on your guess. The remaining term is nonnegative, and becomes zero at $c=m$. The mean minimizes expected squared loss, assuming the second moment is finite.

Best depends on the loss

Suppose the three equally weighted outputs are instead 0, 0, 9. Their mean is 3 and their median is 0.

scoring rule	guess 0	guess 3	best prediction
average squared error	27	18	mean: 3
average absolute error	3	4	median: 0

Absolute loss is a useful alternative when you want less sensitivity to large errors. Squared loss gives large errors extra weight and leads to the decomposition above. The claim “the mean is best” always needs its scoring rule.

Now at every x

Apply the same reasoning to the conditional distribution at each input. The best predictor under squared loss is

$$f(x)=\mathbb E[Y\mid X=x].$$

When a conditional density exists, this average can be written

$$f(x)=\int_{-\infty}^{\infty} y\,f_{Y\mid X}(y\mid x)\,dy.$$

Here $f(x)$ is a prediction, while $f_{Y\mid X}$ names a density. They use the same letter for different objects; the subscript distinguishes them. In the integral, $x$ stays fixed and $y$ runs through the outputs. Multiply each output by its probability density and add over output values.

A compact statement is

$$f=\operatorname*{arg\,min}_{g}\mathbb E[(Y-g(X))^2].$$

Argmin returns the choice that achieves the lowest score. Here the choices are functions $g$. At every input, choosing the conditional mean minimizes the conditional score; averaging those minimal scores over inputs gives the smallest overall expected score.

The floor, and the part above it

For a fixed predictor $h$ and a fixed input $x$, under our constant conditional noise variance assumption:

$$\mathbb E[(Y-h(x))^2\mid X=x]=(f(x)-h(x))^2+\sigma^2.$$

The first term is error from missing the conditional mean. The second is the variation of the output around that mean. Even knowing $f$ exactly leaves expected squared error $\sigma^2$. This is irreducible error, relative to the inputs you have measured. It is a floor for expected error; an individual test sample can score below it by chance.

Take $Y=X^2+\varepsilon$, with independent $\varepsilon\sim\operatorname{Unif}[-1,1]$. At $x=1/2$:

• $Y=1/4+\varepsilon$: uniform on $[-3/4,5/4]$.
• Its mean is $1/4$ and its variance is $1/3$.
• A rival $h(x)=x^3$ guesses $1/8$.

$$\begin{aligned} \mathbb E[(Y-h(x))^2\mid X=1/2] &=(1/4-1/8)^2+1/3\\ &=1/64+1/3=67/192. \end{aligned}$$

Work through this style of question, from the original notation onward.

Practice prediction error

Next

You know what the best predictor is in theory. How do you estimate it when you only have a finite dataset?