The Math Beneath · Part 8 / 8

Downhill is a direction

The derivative is a purely local measurement — which way is down, and how steep. Following it, step after step, is all of training. The learning rate is your stride.

Chapter seven gave every setting of a model’s weights a price — a loss. Picture that as a landscape: each position is a choice of weights, the height is the loss there. Training is finding a low point.

Here’s the constraint that shapes everything: you can’t see the valley. The landscape has too many dimensions to survey, and computing the loss everywhere is the whole-world problem again. All you can do is stand where you are and feel the ground under your feet.

The slope under your feet

That local feeling is the . Zoom in on any smooth curve far enough and it stops curving — up close, it’s just a straight line. The derivative is that line’s slope: if I nudge this weight a hair, does the loss go up or down, and how steeply? One number, measured entirely at your feet, no map required.

And one number is enough, because it hands you a policy:

w ← w − lr · slope

Step against the slope — downhill — scaled by a stride length lr, the . Then re-measure, step again. That loop is , and it is the entire engine of learning.

Try it: step slowly with a small stride and watch the loss trace sink smoothly. Now crank the learning rate past 1 and run — the ball vaults the valley and climbs the far wall, each “downhill” step landing higher than the last. Then drop the ball on the far left with a small stride: it settles in the shallow dip and stops, slope zero, blind to the lower valley next door.

Three fates, one dial

Everything you just saw generalizes uncannily well to real training:

Crawl. Too small a stride wastes compute; the loss inches down.
Converge. The right stride rides the slope down and settles.
Oscillate or diverge. Too large a stride turns downhill measurements into uphill motion — the spiking loss curves of real training runs are this picture.
The nearest dip. Local steps find a bottom, not the bottom. Where you start, and the noise in your steps, decide which valley you inherit.

Go deeper: millions of dimensions, and the chain rule

With millions of weights, the derivative becomes a list — every weight’s personal slope, bundled into one vector, the gradient. “Step against the slope” is unchanged; it just happens in every dimension at once.

Computing all those slopes cheaply relies on one fact: slopes multiply through composed steps. If a weight nudges a hidden value, and that hidden value nudges the loss, the weight’s slope on the loss is the product of the two local slopes. That’s the chain rule, and is nothing but disciplined bookkeeping for it — sweeping once backward through the network, multiplying local slopes, as the training chapter showed.

One more echo of chapter four: real training measures the slope on a random batch, not the whole world — a noisy sample whose expectation is the true gradient. Noisy downhill still averages out to downhill. That’s stochastic gradient descent, and it’s the only reason any of this fits in a computer.

The whole book in one breath

The log turns multiplies into adds — so beliefs (log-odds), evidence (points), and surprise (log 1/p) all live on straight rulers. Bayes says every walk starts from a prior. Expectation turns randomness into one number you can chase by sampling. Cross-entropy prices your beliefs against the world’s — floor plus gap. And the gradient is how the gap actually closes: one local, downhill step at a time.

Eight moves. Every loss, sigmoid, softmax, KL leash, and training curve in this library is these eight, wearing work clothes.

Sources · 4

Full bibliography →