One dataset, from rows to a final prediction

Here is a small, invented dataset. One input $x$ goes with one observed output $y$. We chose the numbers so you can check the entire calculation by hand.

Training record	Input x	Output y
1	0	0
2	1	3
3	2	4
4	3	3

We also have two validation observations for choosing a model and two test observations for evaluating the final choice. Start with the four training rows.

Start with one prediction for every row

A constant model predicts $c$ everywhere. Under squared error, its best choice is the average training output:

$$c=\frac{0+3+4+3}{4}=\frac52.$$

Its four squared errors are $25/4,1/4,9/4,1/4$. They sum to 9; their average is $9/4=2.25$.

Keep the answers separate: the best constant is 2.5; its training MSE is 2.25. “Argmin” would ask for the first; “minimum loss” would ask for the second.

Inspect each candidate on the same training rows. Open validation when ready, choose the model, then reveal its final test result.

Let the prediction depend on the input

A line predicts $h(x)=a+bx$. The intercept $a$ is added to every prediction; $b$ multiplies the input. Our training loss is

$$L(a,b)=\frac14\left[a^2+(a+b-3)^2+(a+2b-4)^2+(a+3b-3)^2\right].$$

Each term comes from one table row. For instance, record 3 has $x=2,y=4$, so its prediction error is $a+2b-4$.

To fit the line, choose both parameters to minimize that expression. Its partial derivatives are

$$\frac{\partial L}{\partial a}=2a+3b-5,\qquad \frac{\partial L}{\partial b}=3a+7b-10.$$

Setting both equal to zero gives $a=b=1$. You can check by substitution: $2+3-5=0$ and $3+7-10=0$. This squared-error objective is a strictly convex quadratic, so the stationary point is its unique minimum.

The line $1+x$ predicts $(1,2,3,4)$. Observed minus predicted gives $(-1,1,1,-1)$. Every squared error is 1, so training MSE is 1.

Where did the derivatives come from?

For one row, the error is $a+bx_i-y_i$. Differentiating its square gives twice the error. The derivative of the inside is 1 with respect to $a$, and $x_i$ with respect to $b$:

$$\frac{\partial L}{\partial a}=\frac24\sum_i(a+bx_i-y_i),\qquad \frac{\partial L}{\partial b}=\frac24\sum_i(a+bx_i-y_i)x_i.$$

The row totals are $\sum x_i=6$, $\sum y_i=10$, $\sum x_i^2=14$, and $\sum x_iy_i=20$. Substituting gives the two derivatives above. This is the chain rule applied to the actual rows.

Take one gradient step

Solving the equations found the completed fit. A gradient method can approach it through updates.

Start at $(a,b)=(0,0)$, whose training MSE is $34/4=8.5$. The gradient there is $(-5,-10)$. With learning rate $0.1$:

$$(a,b)_{\text{new}}=(0,0)-0.1(-5,-10)=(0.5,1).$$

The new line predicts $(0.5,1.5,2.5,3.5)$ and has training MSE $1.25$. One step improved the fit; it has not yet reached the minimum of 1.

In the explorer, this update is a separate fitting demonstration. The model comparison uses each candidate’s completed fit.

A perfect training fit

The quadratic $q(x)=4x-x^2$ predicts $(0,3,4,3)$, exactly matching the training outputs. Its training MSE is zero.

That alone does not tell us whether it predicts new outputs well. Compare the three fitted candidates using the validation rows.

Work through the validation calculation

The validation observations are $(0,1)$ and $(3,5)$.

Candidate	Predictions	Squared errors	Validation MSE
Constant 2.5	2.5, 2.5	2.25, 6.25	4.25
Line 1+x	1, 4	0, 1	0.5
Quadratic 4x−x²	0, 3	1, 4	2.5

The line has the lowest validation error among these candidates. Two observations provide a small amount of evidence; this ranking is not guaranteed for every future sample.

Freeze the choice, then evaluate it

Choose the line using validation. Keep its parameters fixed, then open the test rows.

The final test calculation

The final test observations are $(1,1)$ and $(2,4)$. The frozen line predicts 2 and 3. Its squared errors are 1 and 1, so final test MSE is 1.

We report that score. If we use it to change the model, these observations become part of selection and no longer provide an untouched test of that new choice.

Connect the measured errors to probability

For this teaching example, we generated possible outputs using

$$X\sim\operatorname{Unif}\{0,1,2,3\},\qquad Y=1+X+\varepsilon,$$

where independent noise $\varepsilon$ is equally likely to be $-1,0,1$. The fitting procedure receives the training rows, not this rule. The displayed rows are selected possible realizations chosen for easy arithmetic.

At each input, the population mean is $1+x$. Our fitted line happens to match it exactly in this constructed sample. Its expected squared error is therefore

$$\mathbb E[\varepsilon^2]=\frac{1+0+1}{3}=\frac23.$$

Training MSE was 1, validation MSE was 0.5, and test MSE was 1. Those scores used a few particular observations. The expectation averages all possible noise outcomes with their probabilities. Different numbers are expected here.

The same calculation in matrix notation

Put an intercept column of ones beside the inputs:

$$A=\begin{bmatrix}1&0\\1&1\\1&2\\1&3\end{bmatrix},\quad \theta=\begin{bmatrix}a\\b\end{bmatrix},\quad A\theta=\begin{bmatrix}a\\a+b\\a+2b\\a+3b\end{bmatrix}.$$

The output vector is $y=[0,3,4,3]^\mathsf T$. The squared Euclidean norm $\lVert A\theta-y\rVert^2$ adds the squares of its four entries. Then the loss is $\frac14\lVert A\theta-y\rVert^2$. It still makes four predictions, subtracts the four outputs, squares the errors, and averages them. Linear algebra explains how to read each shape and product.

Check your understanding

Why is the line’s validation error below the expected noise variance? Why did we choose the line even though the quadratic’s training error was zero?

Compare the experiments being measured

The validation error uses two realized outputs; the noise variance averages all possible outputs at a fixed input. A finite sample can lie below that expectation. The quadratic fits the training sample, but the line performed better on the separate validation observations used to choose the model.