How it learns

Next-token targets, loss, and parameter updates

The weights are learned from examples. A training step computes a loss and uses its derivatives to update parameters. Generation normally uses those parameters without changing them.

Guess, check, nudge

Take a token sequence from the training data. At each position, predict the next token using only the tokens up to that position. The observed next token is the target.

Check — the loss is the negative log of the probability it gave the target token: low if that probability was high.
Update — use gradients of the batch loss to change the parameters. One update need not improve every example.

Backpropagation calculates derivatives of the loss. An optimizer uses them to choose updates. A basic gradient-descent update subtracts a learning rate times the gradient; actual training need not reduce the loss on every step.

see this in PyTorch

Assume model is a causal language model and optimizer is already configured. tokens has shape (batch, length) and contains equal-length examples with at least two tokens. Shift each example by one position to construct its targets.

import torch.nn.functional as F

inputs = tokens[:, :-1]          # everything except the final token
targets = tokens[:, 1:]          # the token after each input position
logits = model(inputs)           # (batch, length - 1, vocabulary_size)

# cross_entropy expects one row of scores and one target ID per example.
# Here each token position becomes one example; the loss averages them.
loss = F.cross_entropy(
    logits.reshape(-1, logits.size(-1)),
    targets.reshape(-1),
)

optimizer.zero_grad()            # clear gradients from the previous step
loss.backward()                  # compute derivatives of the batch loss
optimizer.step()                 # update parameters using those gradients

For the sequence [the, cat, sat], inputs are [the, cat] and targets are [cat, sat]. Causal attention prevents each prediction from seeing its target. Padded batches also need attention masks and a loss that ignores padding.

Go deeper: which way is downhill? (backprop)

“Nudge” means stepping downhill on a hill of error. Gradient descent steps the opposite way the slope points; backprop computes that slope for every weight in one backward pass.

Go deeper: how it becomes the assistant you chat with

A pretrained model can already answer some questions, but its training objective does not directly specify the assistant behavior we want. Instruction tuning trains on demonstrations; preference methods such as RLHF or DPO use comparative feedback. These are possible post-training stages, not a mandatory recipe shared by every model.

Some training tasks supply verifiable feedback, such as a checked answer or passing program tests. Such feedback can reward useful solution behavior. A written reasoning trace is still not a guaranteed faithful account of the internal computation.

Next: how scaling, caching, and low-rank updates change resource costs.

Sources · 6

Full bibliography →