Machine LearningModule 3 · Guide and practice
Module 3 · Fit a line and know what it means
Multiply inputs by weights. Add a baseline. Check the misses. Check the uncertainty. That is the path through Module 3.
Read each small idea, then cover the answer and try its calculation. Jump to Q1: matrices, Q2: averages, Q3: fit a line, or Q5: test slopes. Keep the formula sheet nearby.
Browse the original slides above each topic; click a slide to enlarge it. Practice-sheet previews include the original solutions.
Matrix = a grid of numbers
Section titled “Matrix = a grid of numbers”
- Scalar: one number.
- Vector: an ordered list, usually a column.
- Matrix: a rectangular grid.
- : matrix has rows, columns, and real entries.
- : the entry in row , column .
- Transpose : swap rows and columns.
For example, is the column with 2 above 1.
Multiplication = row meets column
Section titled “Multiplication = row meets column”Shape trick: inner dimensions match; outer dimensions survive.
Multiply matching entries, then add:
runs along row of and down column of . The sum sign means “add these terms.”
Invented example: use each row of with input column .
Inner product = number; outer product = grid
Section titled “Inner product = number; outer product = grid”For column vectors :
Invented values: , .
- Inner: .
- Outer: every entry of multiplies every entry of :
Other matrix rules to recognize
- Identity : ones on the diagonal, zeros elsewhere; when sizes match.
- Order matters: usually .
- Grouping works: .
- Distribution works: .
- Transpose twice: .
- Transpose a sum: .
- Transpose a product: — reverse the order.
Practice 1: multiply matrices
Section titled “Practice 1: multiply matrices”View the original practice sheet (includes solutions)
Given: calculate these three products. Rule: matrix multiplication.
Hint: and ; these powers do not square or cube entries individually.
Work through all three products
(a) Use row times column. The four entries are
So the answer is , as expected when multiplying by the identity.
(b) Use the rows and columns of the given matrix twice.
Thus .
(c) Call the given matrix . First multiply by itself:
Then multiply those calculated entries by the original :
Thus .
Eigenvector = an arrow that stays on its line
Section titled “Eigenvector = an arrow that stays on its line”
A matrix can turn an arrow. An eigenvector is a nonzero arrow that the matrix only scales:
- : a square matrix.
- : the eigenvector.
- : the eigenvalue, or scaling factor.
- Negative : reverse the arrow. Zero : collapse it to zero.
Check trick: multiply, then compare.
For invented and :
So is an eigenvector and .
Symmetric matrix: . A real symmetric matrix has real eigenvalues and admits mutually perpendicular eigenvectors. Perpendicular means for different chosen vectors; general real matrices can have complex eigenvalues/eigenvectors.
Invented self-check: with the same , is an eigenvector? Rule.
Check your answer
. A single multiplier cannot turn into : the first entry needs 3 and the second needs 2. So this is not an eigenvector.
Coefficient = multiplier
Section titled “Coefficient = multiplier”
In a line, a coefficient tells you how much to add per input unit:
- : input; : observed output.
- : slope, the multiplier on .
- : intercept, the starting amount.
- : noise, the part the rule does not explain.
Intercept = baseline; hat = estimated
Section titled “Intercept = baseline; hat = estimated”The intercept is the prediction when inputs equal zero. We do not know nature’s coefficients; fitting data gives estimates, marked with hats:
identifies one observation. Uppercase are random variables; lowercase are recorded values. The prediction leaves out unknown noise.
Invented fitted rule: . At :
- Input contribution: .
- Add baseline: .
One extra unit adds 3; at , the prediction is 10.
Multiple regression = more multipliers
Section titled “Multiple regression = more multipliers”With inputs, add one contribution per input:
selects an input. is its per-unit effect holding the other inputs fixed. excludes the intercept.
The data-generating story is “draw inputs, then an output given those inputs”:
is the input density; is the output density given an input; is their joint density. The linear rule and noise distribution describe that conditional output.
Least squares = smallest total squared miss
Section titled “Least squares = smallest total squared miss”
Residual = observed − predicted. For observation :
counts observations. RSS means residual sum of squares. Least squares chooses coefficients that make RSS smallest.
Why square? Invented misses and should both count: , rather than canceling.
Slope recipe = center, multiply, add, divide
Section titled “Slope recipe = center, multiply, add, divide”For observed pairs , first calculate the means:
Then subtract those means — center the values — and build two totals:
- : how inputs and outputs move together.
- : how much the inputs spread out.
Divide to get the slope; use the means to position the line:
Require . Identical inputs cannot tell you the slope.
Memory cue: the intercept puts the line through its center, .
Practice 2: why the line crosses the averages
Section titled “Practice 2: why the line crosses the averages”View the original practice sheet (includes solutions)
Given: the least-squares formulas above. Show that the fitted line passes through . Rule: simple regression.
Hint: substitute into the prediction; replace the intercept with its formula.
Work through the cancellation
The prediction rule is . At the average input, use and :
The two slope terms cancel. The slope itself does not become zero. This property applies to ordinary least squares with an intercept.
TSS = the average-only guess’s error
Section titled “TSS = the average-only guess’s error”
Before fitting a line, predict the sample mean for everyone. Its total squared miss is total sum of squares:
RSS grades the fitted model. TSS grades the average-only baseline.
R² = fraction of baseline error removed
Section titled “R² = fraction of baseline error removed”For example, means 94% less squared error than guessing the mean. It is not the percentage of individually correct predictions.
On training data, least squares with an intercept gives , provided TSS .
RSE = scale of the misses
Section titled “RSE = scale of the misses”Residual standard error estimates noise size in output units.
- : observations.
- : predictors, excluding the intercept.
- : observations minus fitted coefficients, called residual degrees of freedom.
Require and linearly independent model columns: no column is an exact combination of others.
Practice 3: fit and score a line
Section titled “Practice 3: fit and score a line”View the original practice sheet (includes solutions)
Given: with normal noise of mean zero and variance . The three observed pairs are , , and .
Find , , RSS, RSE, TSS, and . Fitting formulas · Scoring formulas.
Hint: follow this order: averages → centered totals → coefficients → predictions → squared residuals → scores. Normality is not needed to do the least-squares arithmetic.
Work through the complete calculation
1. Calculate the averages from the three given observations.
2. Calculate the slope using centered totals. With calculated means , :
The centered outputs come from subtracting from the given outputs . Therefore
3. Calculate the intercept. Use , , and the calculated slope :
The fitted rule is .
4. Predict, subtract, and square. Apply at each given input; calculate :
Add the calculated squared residuals:
5. Estimate noise size. There are observations and predictor. Use calculated RSS :
6. Calculate the mean-only baseline. Use given outputs and calculated :
7. Calculate the relative improvement. With RSS and TSS :
Two quick checks: the calculated residuals sum to , and RSS TSS. Both hold for training-set least squares with an intercept.
Matrix fitting = all predictions at once
Section titled “Matrix fitting = all predictions at once”
The design matrix stores the recipe’s inputs:
- One row per observation.
- One column per predictor.
- An initial column of ones to add the intercept every time.
holds observed outputs; holds coefficients . Predictions are .
is a trial coefficient vector. Argmin = coefficients giving the smallest error. The squared length adds squared entries of residual vector .
With linearly independent columns, the reusable solution is
An inverse undoes a matrix: . The displayed inverse exists when no design column is an exact combination of others.
Same Q3 data, using the matrix formula
Given pairs: . Put ones beside their inputs:
Take column dot products to form :
- Ones with ones: .
- Ones with inputs: .
- Inputs with inputs: .
Multiply columns by outputs to form :
The diagonal matrix’s inverse has entries . Substitute into the fitting rule:
Same intercept, 1; same slope, 3.5.
Standard error = how much the estimate bounces
Section titled “Standard error = how much the estimate bounces”
Collect a fresh dataset and refit. Noise changes, so the estimated weights change.
- True coefficient: fixed but unknown.
- Coefficient estimate: varies between samples.
- Standard error (SE): estimated standard deviation of those estimates.
Smaller SE = a more precise estimate.
Invented self-check: two fits estimate the same slope, 2. Fit A has SE ; Fit B has SE . Which is more precise? Rule.
Check your answer
Fit A. Its relative SE is : its estimated repeated-sample spread is one quarter as large.
RSE and SE describe different spreads
Section titled “RSE and SE describe different spreads”- RSE: scale of output misses.
- Coefficient SE: uncertainty in a fitted multiplier or intercept.
Assumptions for these uncertainty formulas: correct linear model; zero-mean noise given inputs; independent errors of common variance ; linearly independent design columns.
For one predictor, define input spread . If noise standard deviation is known:
Here SD means standard deviation. If variance is supplied, take its square root to get .
Unknown noise? Use RSS → RSE → slope SE:
Replace with RSE in the intercept formula too. Supplied SE? Use it directly.
More noise increases uncertainty. More input spread reduces slope uncertainty — widely separated points reveal the tilt more clearly.
Invented calculation: , , , , RSS .
- Estimate noise: .
- Slope SE: .
- Intercept SE: .
Invented self-check: a fit reports RSE and slope SE . Which number describes uncertainty in the slope? Rule.
Check your answer
describes estimated slope spread across repeated samples. The RSE of 8 describes estimated noise size in response units.
Unbiased = centered on the true coefficient
Section titled “Unbiased = centered on the true coefficient”Under the assumptions above, conditional on the observed design :
- : average over repeated fits. Unbiased means this average hits the true coefficient.
- : covariance matrix, describing how estimates vary together.
- Diagonal entries: coefficient variances. Take square roots for standard deviations.
Unbiased does not mean every estimate is close. SE tells you about spread around that center.
Confidence interval = estimate ± uncertainty margin
Section titled “Confidence interval = estimate ± uncertainty margin”
An approximately normal estimate gives the quick approximately 95% interval:
selects the coefficient. Calculate margin = , then subtract/add it.
Q5(c) supplied values: estimate 2, SE .
- Margin: .
- Lower endpoint: .
- Upper endpoint: .
- Interval: .
Meaning: about 95% of intervals made this way across repeated samples contain the fixed true coefficient. This particular interval is not guaranteed to contain it.
Approximation: 2 rounds the normal cutoff 1.96. With normal errors and estimated noise, exact regression intervals use a Student cutoff with degrees of freedom. The practice uses the stated rule.
t = gap ÷ uncertainty
Section titled “t = gap ÷ uncertainty”
A gap matters more when the estimate is precise. Measure the gap in standard-error units:
- : estimate.
- : proposed true value.
- SE: uncertainty in the estimate.
- : distance in SEs; ignore direction for a two-sided test.
Approximate decision rule
- : reject the proposed value.
- : fail to reject; insufficient evidence against it.
Same trick, two views: the proposed value is over 2 SEs away exactly when it is outside the interval. Endpoints count as inside in these practice questions.
The formal hypotheses are (null) and (alternative). To test for a linear relationship, test slope .
A nonzero fitted slope can arise from noise even when the true slope is zero. Rejection supports an association under the model; it does not establish causation.
Significance : the long-run rate of rejecting a true null, under the assumptions. It is not the probability the null is true. Failing to reject does not prove it true.
Invented self-check: estimate , SE , proposed slope . Reject under the approximate rule? Rule.
Check your answer
Gap: . Divide by SE: . Since , reject. Interval check: margin , giving , which excludes 0.
Practice 5: test two proposed slopes
Section titled “Practice 5: test two proposed slopes”View the original practice sheet (includes solutions)
Given: . Test the two separate claims and in each case below. Rule: confidence intervals and tests.
| Case | Given | Given SE |
|---|---|---|
| (a) | ||
| (b) | ||
| (c) |
What this asks: could the true slope be 1? Could it be 2? Test each value separately. Here and are names for two separate claims, not a null/alternative pair.
Hint: two claims, two distances. Calculate (estimate − claim) ÷ SE separately for each claim. Reject only past 2 in magnitude. The interval provides a second check.
Work through the three decisions
(a) Estimate , SE . Use the gap divided by SE for each claim:
The claims are 4 and 5 SEs away. Both distances exceed 2, so reject and reject . Check with the interval: its margin is ,
Both 1 and 2 are outside that range.
(b) Estimate , SE . Divide each gap by the supplied SE, 2:
The claims are only 0.25 and 0.75 SEs away. Fail to reject either claim. The estimate is too uncertain to rule either out. The interval check uses margin :
Both 1 and 2 are inside. This does not establish that both values are true; it means the data do not distinguish them well enough.
(c) Estimate , SE . Divide each gap by the supplied SE, :
The first claim is exactly 2 SEs away; the second matches our estimate. Fail to reject either under the stated rule, which rejects only distances greater than 2. The interval has margin :
The value 1 is on the boundary and 2 is inside. The boundary decision depends on the chosen approximation.
What to do when a new problem changes the numbers
Section titled “What to do when a new problem changes the numbers”- Matrix: inner shapes match; outer shapes survive. Row times column.
- Eigenvector: multiply; check for one common scaling factor.
- Line: center → multiply → add → divide; intercept puts it through the averages.
- Fit: RSS grades the line; TSS grades the mean; R² compares; RSE measures noise size.
- Uncertainty: SE measures estimate spread; is gap divided by SE.
Next: Module 4 turns categories, interactions, and curves into inputs for the same weighted recipe.