Dots with a hidden rule

Regression functions and noise

Suppose you know how much a campaign spent on advertising. You want to predict its sales. Before choosing a model, we need to say what a prediction is trying to estimate.

What you are handed

Here are some rows from a dataset of 200 campaigns. Advertising spends are in thousands of dollars; sales are in thousands of units.

campaign	TV	radio	newspaper	sales
1	230.1	37.8	69.2	22.1
2	44.5	39.3	45.1	10.4
3	17.2	45.9	69.3	9.3

The three spends are the inputs, also called features or predictors. Sales are the output, also called the response or target.

notation	what it refers to here
$X=(X_1,\ldots,X_p)$	the input for a randomly selected campaign
$x=(230.1,37.8,69.2)$	one particular input
$Y$ ; $y=22.1$	the random output; one observed output
$p=3$	number of features in each row
$N=200$	number of rows

A subscript can identify a feature or a row. Here $x_j$ means feature $j$ of one input; later $(x_i,y_i)$ will mean training example $i$ . Check what the author is indexing before doing any arithmetic.

Same input, different output

Identical advertising spends can produce different sales. The inputs leave out weather, competitors, and other influences. Imagine repeating a campaign with the same input $x$ : the possible sales form a conditional distribution, $Y\mid X=x$ .

Define $f(x)$ as that distribution’s mean:

f(x)=\mathbb E[Y\mid X=x].

Read from the inside out: fix the input at $x$ , consider the possible outputs $Y$ , and average them. The answer is one number for this input. Repeating the calculation at different inputs defines a function.

This is the regression function. It describes average sales associated with the input. It does not, by itself, prove what would happen if you intervened and changed the advertising budget.

Define the leftover; then state the assumptions

For each outcome, subtract its conditional mean. Call the leftover $\varepsilon$ (epsilon):

\varepsilon=Y-f(X),\qquad Y=f(X)+\varepsilon.

For example, at an input whose mean sales are 13, outputs 12.6, 13.9, and 12.5 have leftovers −0.4, +0.9, and −0.5. These three happen to average to zero. A small random sample need not do so exactly; the population of possible leftovers averages to zero at every input, by the definition of $f$ :

\mathbb E[\varepsilon\mid X=x]=0.

That conditional statement is stronger than merely saying the leftovers average to zero when all inputs are mixed together.

For the examples in this module, we additionally assume

\operatorname{Var}(\varepsilon\mid X=x)=\sigma^2.

In words: the noise has the same variance at every input. This is constant noise variance, or homoscedasticity. It is a simplifying assumption, not part of the definition of regression. Real data can have more scatter at some inputs than others; then write $\sigma^2(x)$ instead.

Our simulations generate noise independently of $X$ . Independence fixes the whole noise distribution across inputs; constant variance fixes only its spread. Neither independence nor constant variance is needed to define the conditional mean.

The function you know and the function you estimate

In real data you see the rows, not $f$ . A model fitted from them is written $\hat f$ —“f hat.” The hat tells you this is an estimate.

Numerical outputs lead to regression. Categorical outputs lead to classification. Without an output column, grouping similar inputs is one possible unsupervised task, called clustering. This module focuses on regression scored by squared error.

Prediction asks whether $\hat f$ makes useful guesses. Statistical inference asks what the data support about relationships and parameters. Causal conclusions need additional assumptions or experimental design.

We defined $f$ using a mean. Why should the mean be the prediction you want?