principles.fyi · the brain · concept

gradient descent

Take tiny steps downhill to make the model's mistakes smaller.

new_weight = old_weight - (learning_rate × gradient)

Picture the loss as a big bumpy hill, and the model's weights as where you're standing on it. The gradient is an arrow that points the steepest way UP, so you take a tiny step the opposite way — downhill. Each little step nudges the weights so the model is a bit less wrong, and doing this over and over slowly walks you down toward the bottom. The arrow itself — how each weight should change — is worked out by backpropagation (one backward pass of the chain rule); gradient descent is just the rule that takes that arrow and actually moves the weights. Together, that's how almost any neural network, transformers included, learns.