principles.fyi · the brain · concept

LoRA (low-rank adaptation)

Keep a frozen brain; learn a tiny side-note made of two skinny grids.

W_new = W + A·B, rank r ≪ size (A is tall-skinny d×r, B is wide-skinny r×k; only A and B are trained, W is frozen)

The model's big weight grid is huge and expensive to retrain, so LoRA leaves it frozen and learns a small change on the side, called the update (ΔW). Instead of storing that full-size update, LoRA builds it from two skinny grids, A and B; multiplied together they form a grid the same shape as the big one, but assembled from far fewer numbers. This works because the useful update is low-rank: it can be reconstructed from just a few independent directions, so a couple of skinny columns (in A) and rows (in B) are enough to span it. At run time you add this learned update back onto the frozen weights. You end up training a tiny sliver of the parameters and still capture most of the benefit.