principles.fyi · the brain · concept

learning rate

The stride length of gradient descent — how far to step in the downhill direction.

w ← w - lr · ∇L(w)

Each gradient-descent update moves the weights by minus the learning rate times the slope, so this one dial converts a local measurement into an actual step. Too small and training crawls; about right and the loss rides smoothly down; too large and 'downhill' steps overshoot the valley floor and land higher on the far wall — oscillation, or full divergence when each bounce climbs. Real training schedules the stride (warmup, decay) because the useful step size shrinks as the model settles into a minimum.