Partial derivatives and backpropagation

Backpropagation calculates how a loss changes when each parameter changes. An optimizer then uses those derivatives to update the parameters. This sequence shows both jobs with numbers you can check.

We begin with a prediction of 3 when the observed answer is 5. For the first four lessons, the same two parameters, w and b, stay in view. Then we add a hidden layer and apply the same rules.

You only need basic derivatives and school algebra to begin. Matrix multiplication helps with lesson 5. This belongs in the math foundations; neural-network training is its application.

Six lessons

1. One parameter at a time — Identify what is fixed, calculate a partial derivative, and interpret its sign.
2. From partials to an update — Combine two derivatives in a gradient and update both parameters together.
3. The chain rule, with numbers — Multiply local rates along a path; add contributions when a parameter has several paths.
4. Work backward through a calculation — Trace values forward, then loss derivatives backward, one operation at a time.
5. A small network, by hand — Follow nine parameters through a hidden layer, ReLU, and the loss.
6. What backprop does—and what it costs — Connect the calculation to automatic differentiation and the training loop.

Each lesson includes a question with a worked answer. You can try it before opening the explanation.

When the lecture uses matrices

Optional reference: Jacobians and vector–Jacobian products translates the same chain rule into matrix notation. Read it after the backward calculation makes sense; it is not required to follow the six lessons.

Sources collect the notes and papers used here.