From scores to a token

Softmax, temperature, and decoding

One logit per word. Two steps left: make them probabilities, pick one. Then close the loop from Chapter 1.

Softmax: scores → probabilities

Softmax exponentiates the scores and divides by their sum:

p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}.

For scores $(0,\ln2,\ln3)$ , the exponentials are $(1,2,3)$ , so the probabilities are $(1/6,2/6,3/6)$ . A score difference of 0.1 gives a probability ratio of $e^{0.1}\approx1.105$ ; a slightly larger score need not dominate.

Temperature changes how concentrated the probabilities are

Divide the logits by a temperature first:

Low (0.3): more concentrated on the highest-scoring tokens.
High (1.5): more spread out, giving lower-scoring tokens more probability.

see this in PyTorch

Take the model’s next-token scores, adjust their sampling distribution with temperature, turn them into probabilities, then either roll a weighted die (sampling) or always pick the top one (greedy).

# logits: the model's raw scores for the next token  # logits: (vocab,)
import torch

temperature = 0.8  # <1 = more concentrated, >1 = more spread out, 1 = unchanged

scaled = logits / temperature           # squash or spread the scores  # (vocab,)
probs = torch.softmax(scaled, dim=-1)   # turn scores into probabilities that sum to 1  # (vocab,)

# sample: roll a weighted die, so likely tokens win more often (but not always)
next_token = torch.multinomial(probs, num_samples=1)  # (1,)

# greedy alternative: always grab the single most likely token (no randomness)
next_token_greedy = torch.argmax(probs, dim=-1)  # ()

Trimming the field: top-k & top-p

Sampling rules can restrict the candidate set before drawing a token. Renormalize the probabilities of the retained candidates so they sum to 1:

Top-k — keep the k likeliest words.
Top-p — sort by probability and keep the smallest leading set whose total reaches or exceeds p.
Min-p — keep tokens at least a fraction (say 5%) as likely as the top word. The bar rises with the model’s confidence.

Close the loop

Choose a token, such as on, and append it to the sequence. The next prediction conditions on that longer sequence. A straightforward implementation recomputes all positions; a KV cache reuses earlier keys and values. The new token still passes through the model’s layers.

Temperature changes the sampling distribution, not the trained weights or the truth of an answer. A low-temperature result can be confidently wrong. Greedy decoding chooses a highest-scoring token directly; it does not require dividing by a temperature of zero.

see this in PyTorch

To generate text, the model keeps reading everything written so far, guesses the next token, sticks it on the end, and runs again.

import torch

# `model` maps a sequence of token ids -> logits for every position.
# logits[..., t, :] = scores over the whole vocab for "what comes after position t".

tokens = torch.tensor([[1, 14, 27]])   # our prompt so far: (batch=1, seq=3)

for _ in range(5):                      # generate 5 new tokens, one at a time
    logits = model(tokens)              # run model on ALL tokens so far: (1, seq, vocab)
    last_logits = logits[:, -1, :]      # we only care about the LAST position: (1, vocab)
    probs = torch.softmax(last_logits, dim=-1)   # turn scores into probabilities
    next_token = torch.multinomial(probs, num_samples=1)  # sample one token: (1, 1)
    tokens = torch.cat([tokens, next_token], dim=1)       # append it -> seq grows by 1

# after the loop, `tokens` holds prompt + 5 freshly generated tokens
print(tokens)

That’s the core machine

text → vectors → attention + FFN ×N → last vector → logits → softmax → sample → append → repeat

N is the model’s chosen number of blocks. The layers process numbers; the decoding rule turns their final scores into a token choice.

Three things left: where the weights came from (training), how models get so big (scale), and whether we can peek at what they learned.

Sources · 4

Full bibliography →