Read the question, then do the math

A question can look unfamiliar because several familiar operations have been packed into one line. Start by identifying the objects, the assumptions, and the requested output. Try each problem before opening its walkthrough.

A conditional-error question

Question. Let $Y=X^2+\varepsilon$, where $\varepsilon\perp X$ and $\varepsilon\sim\operatorname{Unif}[-1,1]$. For $h(x)=x^3$, compute

$$\mathbb E[(Y-h(X))^2\mid X=1/2].$$

You are being asked for one number. The condition after the vertical bar fixes $X$. The outer expectation averages the squared error over the remaining randomness in $Y$.

1. Decode what you were given

• $\varepsilon\perp X$: epsilon is independent of X. Fixing X does not change its distribution.
• $\varepsilon\sim\operatorname{Unif}[-1,1]$: the noise is uniform on that interval. Its mean is zero and variance is $1/3$.
• $h$: the prediction rule you must evaluate, not the true conditional mean.
• $\mathbb E[\cdot\mid X=1/2]$: set X to one-half, then average over the noise.

2. Substitute before expanding

$$Y=1/4+\varepsilon,\qquad h(1/2)=1/8.$$

So the random error is $1/8+\varepsilon$. The question has become: what is the average square of one-eighth plus this noise?

3. Square and average

$$\begin{aligned} \mathbb E[(1/8+\varepsilon)^2] &=1/64+(1/4)\mathbb E[\varepsilon]+\mathbb E[\varepsilon^2]\\ &=1/64+0+1/3=67/192. \end{aligned}$$

Why does $\mathbb E[\varepsilon^2]=1/3$? Variance is $\mathbb E[\varepsilon^2]-(\mathbb E[\varepsilon])^2$, and the mean is zero.

Check the answer: it is above $1/3$, the expected error of guessing the conditional mean. The extra $1/64$ is the squared miss of our rival prediction.

An argmin question

Question. For observations $y_1=0$, $y_2=0$, and $y_3=9$, find

$$c^*=\operatorname*{arg\,min}_{c\in\mathbb R}\frac13\sum_{i=1}^{3}(y_i-c)^2.$$

Decode the question

$c\in\mathbb R$ means you may choose any real number. The sum scores that choice on the three observations. Argmin asks for the winning choice of c, not its score. The star labels the optimal choice; it is not multiplication.

Solve and distinguish the two answers

The mean minimizes squared error, so $c^*=(0+0+9)/3=3$.

The minimum score is $(9+9+36)/3=18$. An answer of 18 would answer “what is the minimum?” rather than “which c achieves it?”

Change the loss to $\lvert y_i-c\rvert$ and the optimal choice becomes the median, 0. Always read the scoring rule.

An expectation over training datasets

Question. At $x_0$, $f(x_0)=4$, $\mathbb E_D[\hat f_D(x_0)]=5$, $\operatorname{Var}_D(\hat f_D(x_0))=2$, and $\operatorname{Var}(Y_0\mid X_0=x_0)=3$. Assume independent training and future examples. Find the expected squared prediction error at $x_0$.

What is random this time?

The input $x_0$ is fixed. The dataset subscript D means we repeatedly draw training datasets, fit a model to each, and inspect its prediction at $x_0$. There is also fresh output noise. The variance of the predictions (2) and the variance of the outputs around f (3) refer to different experiments.

Work the three terms

Bias is $5-4=1$. Squared bias is 1. Add prediction variance and noise variance: $1+2+3=6$.

This is a theoretical average over datasets and fresh outputs. It does not say that the MSE on your next finite test set will equal 6.

A vector calculation

Question. Let $x=(2,5)$, $\beta=(3,-1)$, and $\beta_0=4$. Evaluate $\beta_0+\beta^\top x$.

Translate and calculate

The transpose symbol $\top$ turns the coefficient column into a row so the product is a dot product. Multiply corresponding entries and add:

$$\beta_0+\beta^\top x=4+(3\cdot2)+(-1\cdot5)=5.$$

There are two features in this one example. These are not two training rows.

What does zero mean actually give you?

Question. Let $Y=2X+\varepsilon$, with finite means and $\mathbb E[\varepsilon]=0$. Must $\mathbb E[Y\mid X]=2X$?

Separate the overall average from the conditional average

Conditioning gives

$$\mathbb E[Y\mid X]=2X+\mathbb E[\varepsilon\mid X].$$

The given overall average does not determine the average within each input group. For example, let X be equally likely to be −1 or 1, and let $\varepsilon=X$. The noise has overall mean zero, but $Y=3X$, so $\mathbb E[Y\mid X]=3X$.

We need zero conditional mean, $\mathbb E[\varepsilon\mid X]=0$. Independence of X and ε together with zero overall noise mean would also be sufficient. Zero overall mean alone is not.

Try a new question

Mixed practice changes the numbers and the kind of question. Start without a hint; if needed, reveal a translation before the worked answer.