Use slopes to find minima

Sign

The prediction $f(x + \Delta x) \approx f(x) + f'(x)\,\Delta x$ describes the change for a small step to the right:

• $f'(x) > 0$: $f$ goes up. Rising.
• $f'(x) < 0$: $f$ goes down. Falling.
• $f'(x) = 0$: nothing happens, to first order. Flat.

The size says how fast. A slope of 6 turns a step of $0.01$ into a rise of about $0.06$.

Candidates for maxima and minima

Suppose $f$ has a smooth peak at $x$, and suppose the slope there were positive. Then a tiny step right would go higher. So it wasn’t a peak. Suppose the slope were negative. A tiny step left would go higher. Same problem. At an interior differentiable peak, the slope must be zero. The same reasoning applies to an interior valley.

For a differentiable function, an interior maximum or minimum must satisfy $f'(x)=0$. These points are candidates: you must also check boundaries and any points where the derivative does not exist. For example, find the lowest point of $f(x)=x^2-4x+1$.

$$f'(x) = 2x - 4 = 0 \quad\Longrightarrow\quad x = 2, \qquad f(2) = -3$$

Completing the square proves the global minimum: $f(x)=(x-2)^2-3\ge-3$, with equality at $x=2$. Checking two nearby points alone would not prove a minimum.

A zero derivative does not guarantee a maximum or minimum: $x^3$ has derivative 0 at the origin but keeps increasing through it. A local minimum also need not be the lowest point across the whole domain.

The slope of the slope

If $f'$ is differentiable, its derivative $f''$, “f double prime”, measures how quickly the slope changes. If $f$ is position, $f'$ is velocity and $f''$ is acceleration.

At a flat spot, $f''$ tells hill from valley:

• $f'' > 0$: the slope is increasing, from negative through zero to positive. Falling, then rising. A valley.
• $f'' < 0$: rising, then falling. A hilltop.
• $f'' = 0$: no verdict.

For $x^2 - 4x + 1$, $f'' = 2 > 0$ everywhere. One bowl, and $x = 2$ is its bottom.

f(x) = x⁴/4 − x²: two valleys and a hilltop. Drag the point to read f′ and f″, then step downhill.

Try it

Drag to each flat spot and read $f''$: negative at the hilltop, positive in both valleys. Then start at $x = 2.3$ and press step downhill a few times. Start at $x = -0.3$ and you land in the other valley.

Walk downhill

Solving $2x-4=0$ gives the minimum directly. For a neural network, solving all the corresponding equations is usually impractical. We can instead use the current derivative to choose a direction: a positive slope points downhill to the left, and a negative slope points downhill to the right.

$$x_{\text{new}} = x - \eta\, f'(x)$$

$\eta$ (“eta”) is the learning rate: it scales the derivative to determine the step. The approximation $f(x-\eta f')\approx f(x)-\eta(f')^2$ predicts a decrease when the derivative is nonzero and the step is sufficiently small. Repeating this can approach a local minimum; the result depends on the function, starting point, and learning rate. Here are the first steps from $x=5$ on $x^2-4x+1$, using $\eta=0.1$:

step	$x$	$f'(x) = 2x - 4$	next $x$
0	5	6	4.4
1	4.4	4.8	3.92
2	3.92	3.84	3.536
3	3.536	3.072	3.229

Heading to 2, in shrinking steps. This is gradient descent in one dimension.

Go deeper: how big a step?

A downhill direction does not make every step downhill. On $x^2-4x+1$ from $x=5$, $\eta=1$ lands at $x=-1$, at the same height as the starting point; $\eta=1.1$ lands higher. For a quadratic with constant curvature $f^{\prime\prime}=c>0$, fixed steps converge to its minimum when $0<\eta<2/c$. Other functions need their own checks: curvature may change along the step. This is why the learning rate matters.

Next

A position function gives velocity when differentiated. The next lesson uses a rate over an interval to calculate an accumulated change.