Partial derivatives and gradients

A surface has no single slope

A function can depend on several inputs. For $f(x,y)=x^2y+y$, changing x and changing y generally have different effects. A partial derivative measures one of those effects while holding the other input fixed.

Freeze all but one

Hold every input still except one. That cuts a slice through the surface, and the slice is an ordinary curve.

Stand at $(x, y) = (2, 3)$, where $f = 15$. Freeze $y = 3$: the slice is $f(x, 3) = 3x^2 + 3$, with slope $6x = 12$ at $x = 2$. Freeze $x = 2$: the slice is $f(2, y) = 5y$, a line of slope 5.

These are the partial derivatives, written with a curly $\partial$:

$$\frac{\partial f}{\partial x} = 2xy = 12, \qquad \frac{\partial f}{\partial y} = x^2 + 1 = 5 \qquad \text{at } (2, 3)$$

The $\partial$ notation means: “every other input was treated as a constant.” Differentiate as usual; the frozen inputs ride along like numbers. The common slip is to forget they are there. The $x$-slope of $x^2 y$ is $2xy$, not $2x$.

Check with an actual input change: $f(2.01, 3) = 15.1203$. Predicted: $15 + 12 \times 0.01 = 15.12$.

The surface f(x, y) = x²y + y as a shaded map. Drag the point: the two slices through it are drawn alongside, each with its slope.

Try it

Snap to $(2, 3)$ and read the two slopes, 12 and 5. Drag straight up: the $x$-slice steepens while the $y$-slice, a straight line of slope $x^2 + 1$, does not change.

Small changes add

Change both inputs, $x$ by $\Delta x$ and $y$ by $\Delta y$. For a differentiable function, evaluate both partial derivatives at the starting point. Their contributions add to give the first-order prediction:

$$\Delta f \;\approx\; \frac{\partial f}{\partial x}\,\Delta x + \frac{\partial f}{\partial y}\,\Delta y$$

A differentiable surface is locally approximated by a plane, tilted by the two partials. Two numbers predict the change in every direction. At $(2, 3)$ with $\Delta x = \Delta y = 0.01$: predicted $0.12 + 0.05 = 0.17$. Actual: $f(2.01, 3.01) - 15 = 0.170701$.

The arrow that points uphill

Stack the partials into a vector, the gradient:

$$\nabla f = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}\right) = (12, 5) \quad \text{at } (2, 3)$$

The prediction above is a dot product: $\Delta f \approx \nabla f \cdot (\Delta x, \Delta y)$. A dot product is largest when the two vectors point the same way. For steps of a fixed length, the first-order predicted rise is largest along the gradient.

• The gradient points steepest uphill. Its length, $\sqrt{12^2 + 5^2} = 13$ at $(2, 3)$, is that steepest slope.
• Against it is steepest downhill.
• Perpendicular to a nonzero gradient, the directional derivative is zero. This direction is tangent to the contour through the point; a finite straight step may still change the function.
• For comparison, east 3 and north 4 make a gradient $(3, 4)$ of length 5. Northeast is the unit step $(\tfrac{1}{\sqrt2}, \tfrac{1}{\sqrt2})$, so the slope is $\tfrac{3 + 4}{\sqrt2} \approx 4.95$. More than either axis, less than the 5 available along the gradient.

The same surface as contour lines. The amber arrow shows the gradient’s direction. Swivel the teal probe: its directional derivative is largest along the arrow and zero when tangent to the contour.

Try it

Snap to $(2, 3)$, then choose Align with gradient: the slope reads 13. Choose Perpendicular: the slope reads 0. These controls use the exact directions; the displayed angle is rounded.

Flat spots, and walking downhill

At an interior local minimum of a differentiable function, every slice is flat, so $\nabla f = 0$. That is setting the derivative to zero with many inputs. When solving for the minimum directly is impractical, we can try a gradient step:

$$(x, y)_{\text{new}} = (x, y) - \eta\, \nabla f$$

The linear approximation predicts $\Delta f\approx-\eta\,\lVert\nabla f\rVert^2$. When the gradient is nonzero, a sufficiently small step therefore decreases a differentiable loss. This update is gradient descent; neural-network optimizers often use variants of it. Backpropagation computes the gradients those updates need.

Go deeper: the definition, and a warning

Formally, $\dfrac{\partial f}{\partial x}(a, b) = \lim_{h \to 0} \dfrac{f(a + h, b) - f(a, b)}{h}$, with $b$ never moving. One piece of fine print: having both partials does not by itself make the surface a valid linear approximation at that point. A function can have both partial derivatives at a point without being differentiable there. Continuous partial derivatives in a neighborhood of the point guarantee this linear approximation. The polynomial used here satisfies that condition.

Connect this to fitting a model

The complete dataset example applies the same gradient update to an actual squared-error loss. Backpropagation explains how to calculate those derivatives through many operations.

Next

Use the reference to look up notation and rules. In practice, choose a rule, calculate a derivative or integral, and interpret what the answer means.