Assume a shape, then fit it

The line

The simplest shape: the output is a weighted sum of the inputs, plus a starting level.

$f_L(x;\beta)=\beta_0+\sum_{j=1}^{p}\beta_j x_j.$

symbol	say it	for the ad campaigns
$x_1,\ldots,x_p$	the p numbers in one input row — not p different rows	TV, radio, newspaper spends
$\beta_0$	”beta zero”: where the line sits when every input is 0	predicted sales when all advertising inputs are zero
$\beta_j$	change in the model’s prediction per unit of input j, others fixed	predicted sales difference per unit of TV spend
$f_L(x;\beta)$	the model’s output at row x, using coefficients β	predicted sales

This model has $p+1$ parameters: one coefficient per feature, plus the intercept. For the advertising data, that is four parameters regardless of the number of rows. A parametric model uses a fixed-size list of parameters to describe its possible functions. After fitting, $\hat\beta$ contains the estimated coefficients and $\hat f_L$ is the resulting prediction function.

What the shape assumption buys

method	which observations determine the estimate at x	with N = 1,000, p = 10, r = 0.1
average your neighbors	only those within r of x — about N·r^p	0.0000001 expected observations
the line	all N rows, through the shared coefficients	1,000 rows constrain 11 parameters

An observation far from the target input still helps estimate the slope, because a linear model uses the same slope throughout the input space. This lets all 1,000 rows inform the 11 parameters. The assumption also limits what the model can represent: if the conditional mean bends, no choice of coefficients can reproduce it exactly. The hidden rule in these examples has that curvature.

Fitting: choose the parameters

Choose the parameter values that minimize average squared error on the training data:

$\hat\theta=\operatorname*{arg\,min}_{\theta}\frac1N\sum_{i=1}^{N}(y_i-f(x_i;\theta))^2.$

$\theta$ (“theta”) represents the model’s parameters. The sum runs over the N rows. The quantity being minimized is the training MSE — the mean squared error on the data used to fit the model — and $\hat\theta$ is the parameter estimate that minimizes it. The hat belongs to the fitted parameter: before fitting, f(x; θ) describes the whole family; afterward, f̂(x) = f(x; θ̂) is the chosen model.

For a tiny example, take rows (x,y) = (1,2) and (2,3). The candidate line 1+x predicts both exactly, so its training MSE is 0. The candidate 2x predicts 2 and 4, so its MSE is (0²+1²)/2 = 1/2. Argmin asks which parameter settings win this comparison across all allowed candidates. A numerical fitting algorithm carries out that search.

Polynomial models

A polynomial in one input:

$f(x;\alpha)=\alpha_0+\alpha_1x+\cdots+\alpha_d x^d.$

The degree d sets the parameter count, d + 1, and limits how much the curve can bend. A polynomial is nonlinear in x but still linear in its coefficients. d = 1 is the line. Increasing $d$ adds a coefficient and allows additional curvature.

Fit a polynomial to 40 observations. Change its degree to compare the fitted curve and training error, then reveal the rule that generated the data.

Try it

Reveal the rule. At d = 1 the line can’t follow it. Compare d = 3 and d = 20. Which follows broad structure, and which reacts strongly to individual points? Training error can be below the expected noise floor because fitting adapts to these particular outputs. Then click new dots and watch which degrees change the most.

Next

For these nested polynomial families, the minimum unpenalized training error cannot increase with degree. To evaluate predictions, use observations held out from fitting.