What backprop does—and what it costs

Automatic differentiation, optimizer steps, and computation costs

Backpropagation applies the chain rule in reverse through a neural network’s computation. It answers: how does this scalar loss change with each parameter?

It does not choose the loss, choose a learning rate, or update parameters. Those are separate decisions in the training procedure.

The training loop

Stage	What happens
Forward pass	Use the current parameters to calculate predictions and a loss
Backward pass	Calculate loss derivatives using the forward computation
Optimizer step	Use those derivatives to update parameters
Repeat	Calculate a new forward pass at the updated parameters

For a batch, the loss might be the mean of several example losses. Its gradient is the corresponding mean of their gradients. A step that improves the batch average need not improve every example or unseen data.

Why one backward pass helps

Suppose a model has N parameters. A forward finite-difference estimate for each one would require roughly N additional evaluations of the loss, plus a baseline. A central difference uses two perturbed evaluations per parameter.

Reverse-mode automatic differentiation reuses intermediate derivatives. In the small model, the derivative from the prediction to the loss served both w and b. In a larger graph, the same principle saves repeated work along shared dependencies.

For usual differentiable operations with efficient backward rules, computing the scalar loss gradient costs a small multiple of a forward evaluation. The overhead ratio is roughly constant; the total work is not. A larger or more expensive forward computation still leads to a larger or more expensive backward computation. The exact ratio depends on the operations, which gradients are needed, and the implementation.

What needs to be remembered?

A backward rule may need forward inputs or outputs. Multiplication needs the other input; ReLU needs to know which side of zero it was on. Store the required values or recompute them when needed.

Not every intermediate value must always be saved. Activation checkpointing deliberately saves fewer values and recomputes selected parts of the forward pass during backward. It trades computation for memory. The saving depends on the checkpoint arrangement and model; it is not a universal fixed factor.

Parameters, gradients, optimizer state, and temporary workspaces also use memory.

The same small example in PyTorch

This code computes the exact two-parameter example from lessons 1–4. Read each comment as one stage of the calculation.

import torch

w = torch.tensor(1.0, requires_grad=True)
b = torch.tensor(1.0, requires_grad=True)
x, y = 2.0, 5.0

prediction = w * x + b           # 3
loss = 0.5 * (prediction - y)**2  # 2
loss.backward()                 # compute gradients; parameters stay at 1

print(w.grad.item(), b.grad.item())  # -4.0, -2.0

# A basic optimizer step, using both gradients from the old parameters.
with torch.no_grad():
    w -= 0.1 * w.grad
    b -= 0.1 * b.grad

print(w.item(), b.item())        # approximately 1.4, 1.2

requires_grad=True asks PyTorch to track differentiable operations involving these tensors. backward() accumulates gradients. In a repeated loop, clear gradients before the next backward pass unless accumulation is intentional. A configured optimizer provides zero_grad() and step() for these jobs. See the PyTorch autograd notes.

Three questions to check your understanding

Does backward() make the prediction better?

It calculates derivatives. In the code above, w and b remain 1 immediately after backward(). The update occurs inside the no_grad() block. Whether an update improves the chosen objective depends on the update rule and step size.

Why does generation usually need no backward pass?

Ordinary generation uses fixed parameters to calculate predictions. There is no loss-gradient update in that operation. Some other uses of a trained model do request gradients; “inference never uses derivatives” would be too broad.

Does a zero gradient prove the model is good?

No. It can occur at a minimum, a saddle point, or through inactive paths. It also says nothing by itself about whether the loss measures the desired task or whether the model generalizes.

To read the compact formulas in lectures, continue to the optional Jacobian reference. To see a larger application, return to Transformer training. For evaluation beyond one example, use the complete dataset example.

Sources and further reading