A small network, by hand

The two-parameter model used multiplication, addition, and a loss. Now add two hidden units and a nonlinear function. The backward rule stays the same: multiply local derivatives, and add contributions from shared paths.

Read the model before the derivatives

The input is $x=(1,2)$ and the target is y = 1. A hidden unit first calculates a weighted sum z, then applies $\operatorname{ReLU}(z)=\max(0,z)$ to produce h.

$$W_1=\begin{bmatrix}0.6&-0.2\\-0.3&0.5\end{bmatrix},\qquad b_1=\begin{bmatrix}0.1\\0.2\end{bmatrix},\qquad w_2=\begin{bmatrix}1\\-0.5\end{bmatrix},\qquad b_2=0.2.$$

There are nine parameters: four entries of $W_1$, two of $b_1$, two of $w_2$, and the scalar $b_2$. Biases count as parameters too.

The model is

$$z=W_1x+b_1,\quad h=\operatorname{ReLU}(z),\quad \hat y=w_2^\mathsf T h+b_2,\quad L=\tfrac12(\hat y-y)^2.$$

Read the first equation one row at a time if the matrix notation is unfamiliar. For example, the first hidden unit multiplies x₁ by 0.6 and x₂ by −0.2, then adds 0.1.

Forward: nine parameters, one loss

Quantity	Calculation	Value
$z_1$	$0.6(1)-0.2(2)+0.1$	0.3
$z_2$	$-0.3(1)+0.5(2)+0.2$	0.9
$h_1,h_2$	Both z values are positive	(0.3, 0.9)
$\hat y$	$1(0.3)-0.5(0.9)+0.2$	0.05
L	$\tfrac12(0.05-1)^2$	0.45125

Backward: begin at the prediction

The first loss derivative is

$$g_{\hat y}=\frac{\partial L}{\partial\hat y}=\hat y-y=-0.95.$$

A small increase in the prediction would reduce this loss. Use that derivative to get the output-layer parameter derivatives:

$$\frac{\partial L}{\partial w_2} =g_{\hat y}h=(-0.285,-0.855),\qquad \frac{\partial L}{\partial b_2}=g_{\hat y}=-0.95.$$

For example, $\partial L/\partial w_{2,1}=(-0.95)(0.3)=-0.285$. A weight in a weighted sum receives the downstream loss derivative times the input it multiplies.

Continue through the hidden units

The local derivative of $\hat y$ with respect to a hidden value is its output weight:

$$g_h=g_{\hat y}w_2=(-0.95,0.475).$$

The second sign is positive because increasing h₂ lowers the prediction through the weight −0.5. At the current prediction, that moves us farther from the target.

ReLU has derivative 1 for positive z and 0 for negative z. Both current z values are positive, so $g_z=g_h$. At z = 0, ReLU has no ordinary derivative; this example’s calculator uses the common backward convention of 0 there.

For the first hidden unit, multiply $g_{z_1}=-0.95$ by each input. For the second, do the same with $g_{z_2}=0.475$:

$$\frac{\partial L}{\partial W_1} =\begin{bmatrix}-0.95&-1.9\\0.475&0.95\end{bmatrix},\qquad \frac{\partial L}{\partial b_1} =\begin{bmatrix}-0.95\\0.475\end{bmatrix}.$$

That accounts for every parameter. No parameter has been updated yet.

The graph computes this network’s values and derivatives. Use the table to read the numbers; the diagram shows which values feed which units.

What happens when a hidden unit is inactive?

Set the top-left weight to −0.8. Then $z_1=-1.1$ and h₁ = 0. The ReLU derivative is zero, so this example gives zero derivatives for the weights and bias feeding that unit. Its output-weight derivative is also zero because it multiplies h₁ = 0.

A zero contribution through this unit does not erase other paths. The second hidden unit still contributes, and an input or shared parameter could affect the loss elsewhere. “Inactive for this example” also does not mean permanently inactive on every input.

Check a derivative against the loss

Restore the original parameters. Increase only $W_{1,11}$ from 0.6 to 0.601. The new loss is 0.4503005, a change of −0.0009495. The derivative predicts $-0.95(0.001)=-0.00095$.

A finite-difference check compares these calculations. It is useful for catching errors, but a single matching check is not a proof. Steps that are too large include curvature; steps that are too small suffer rounding error. A step across a ReLU corner needs special care.

Now update all nine parameters

Using the original gradient and $\eta=0.1$ gives

$$W_1^{\rm new}=\begin{bmatrix}0.695&-0.01\\-0.3475&0.405\end{bmatrix},\quad b_1^{\rm new}=\begin{bmatrix}0.195\\0.1525\end{bmatrix},\quad w_2^{\rm new}=\begin{bmatrix}1.0285\\-0.4145\end{bmatrix},\quad b_2^{\rm new}=0.295.$$

A fresh forward pass gives z = h = (0.87, 0.615), prediction 0.9348775, and loss approximately 0.00212047. That is an improvement on this one training example, not evidence of performance on new data.

Your turn: interpret a positive derivative

At the original parameters, $\partial L/\partial w_{2,2}=-0.855$ but $\partial L/\partial h_2=0.475$. Why do those signs differ?

Work it through

They ask about different changes. Increasing $w_{2,2}$ makes its negative value less negative. Since h₂ = 0.9 is positive, this raises the prediction toward the target and lowers L.

Increasing h₂ while holding the negative weight fixed lowers the prediction and raises L. The derivatives describe different inputs to the same multiplication.

Next: connect the calculation to automatic differentiation.

Sources and further reading

• Stanford CS231n: backpropagation and local derivatives.
• Baydin et al.: Automatic Differentiation in Machine Learning.
• Full reading list.