Post-training · Part 2 / 7

Show it good answers

The simplest fix — keep training the model on (instruction, response) pairs. Same next-word objective, but now the next words are a good answer.

A pretrained model only knows one trick: predict the likely next word. Ask it a question and it might continue with more questions — that’s what often follows a question on the web. Likely is not the same as good.

So here’s the simplest fix. Don’t change the trick. Change what you feed it.

Same game, better answers

Pretraining played one game: hide the next word in some web text, guess it, nudge the weights. Instruction tuning plays the exact same game — predict the next word, measure the same cross-entropy loss. Nothing about the math moves.

The only thing that changes is the data. Instead of arbitrary internet text, every example is a pair: an instruction, and a hand-picked good response.

Instruction:  Summarize this in one line: ...
Response:     <a clean, correct summary>   ← the words we train it to predict

It’s called supervised fine-tuning because each instruction comes with a known right answer to copy. (You’ll also see it written SFT.)

Try it: run the prompt through the base model first — it drifts. Then switch to the tuned model. Same network shape, same objective; only the training answers were different.

Why it generalizes

If you only tuned on summaries, you’d get a summarizer. The trick is to tune on many different tasks at once — translation, Q&A, sentiment, meal planning, on and on.

Across all that variety, the cheapest pattern for the model to learn isn’t this task or that task. It’s the general move: read the instruction, do what it says. So it follows instructions it has never seen. That generalization is the whole payoff.

And it’s cheap. Pretraining chews through trillions of words. Instruction tuning needs only thousands to a few million pairs and a handful of passes — a rounding error next to pretraining.

Where the pairs come from

You need a lot of good (instruction, response) pairs. Four ways to get them:

1. People write them. Aya collected 503M instructions across 114 languages — 204K of them hand-written by ~3,000 volunteer fluent speakers.
2. Convert old datasets with templates. Decades of labeled NLP data already exist. Take a movie review and its label, drop them into a template like {{review}} How does the reviewer feel about the movie?, and you’ve minted an instruction. (An LLM can reword the templates for variety.)
3. Repurpose annotation guidelines. The instructions once given to human labelers make ready-made prompts.
4. Have an LLM generate them. For example: take harmful questions and generate safe refusals. Mixing in even ~500 such safety examples sharply cut a model’s harmful answers.

Suppose {{premise}} Can we infer that {{hypothesis}}?
Yes, no, or maybe?

One thing it doesn’t do

Instruction tuning makes the model answer. It doesn’t yet make the model prefer the better of two good answers — clearer, kinder, more honest. For that we need the next idea. First, a footnote on how to do this cheaply.

Next: showing it good answers gets you far, but which answer is best? That’s where preference comes in.