From partials to an update

For $\hat y=wx+b$ with x = 2, y = 5, and w = b = 1, the prediction is 3 and the loss is 2. We found two rates:

$$\frac{\partial L}{\partial w}=-4,\qquad \frac{\partial L}{\partial b}=-2.$$

Together they form the gradient: $\nabla L=(-4,-2)$. The entry order follows the parameter order (w, b).

What happens if both parameters change?

For small changes $\Delta w$ and $\Delta b$, the loss change is approximately

$$\Delta L\approx -4\Delta w-2\Delta b.$$

This is a dot product: combine each parameter’s change with its own rate. If both increase by 0.01, the predicted loss change is −0.06. The exact new loss is 1.94045, a change of −0.05955.

The gradient describes local changes. It does not give an exact prediction for an arbitrarily large move.

Choose a step size

Gradient descent subtracts a positive learning rate $\eta$ times each partial derivative. Use the rates at the same starting point for every parameter:

$$\begin{aligned} w_{\rm new}&=1-0.1(-4)=1.4,\\ b_{\rm new}&=1-0.1(-2)=1.2. \end{aligned}$$

Subtracting a negative increases the parameter. With both new values, the prediction is $1.4\cdot2+1.2=4$, and the loss is $\tfrac12(4-5)^2=0.5$. It was 2.

Computing the gradient and taking the step are separate jobs. Backpropagation will compute the gradient; the update rule decides how to use it.

Take a step, read the new loss, and reset before comparing a different learning rate. All changes are calculated from the displayed parameters.

A bigger step can be worse

Reset to w = b = 1 and use $\eta=0.5$. The parameters become (3, 2), the prediction becomes 8, and the loss becomes 4.5. We overshot the target and increased the loss.

A nonzero gradient gives a downhill direction for sufficiently small steps on a differentiable loss. The useful step size still matters.

Why use the opposite of the gradient?

For a step vector $\Delta\theta$, the first-order prediction is $\Delta L\approx\nabla L\cdot\Delta\theta$. Among steps with the same Euclidean length, this is smallest when the step points opposite to the gradient.

For $\Delta\theta=-\eta\nabla L$, the predicted change is $-\eta\lVert\nabla L\rVert^2$, which is negative when $\eta>0$ and the gradient is nonzero. This is a local statement. The Calculus gradient diagram shows the geometry.

Read a lecture-style question

“Given $\theta=(w,b)$ and $\nabla_\theta L=(-4,-2)$, perform one gradient-descent update with $\eta=0.05$.” What are the new parameters and loss, starting from (1,1)?

Translate and calculate

θ is just the list of parameters. $\nabla_\theta L$ lists the loss derivatives with respect to those parameters. Subtract 0.05 times each entry:

$w_{\rm new}=1.2$, $b_{\rm new}=1.1$. The prediction is 3.5; the loss is $\tfrac12(3.5-5)^2=1.125$.

The derivative is (−4,−2), the parameter change is (0.2,0.1), and the new parameter vector is (1.2,1.1). These are three different objects.

Next: calculate the derivatives without expanding the whole loss.

Sources and further reading

• Stanford CS231n: backpropagation and local derivatives.
• Baydin et al.: Automatic Differentiation in Machine Learning.
• Full reading list.