Judging beats writing

People struggle to write the perfect answer but easily judge which of two answers is better. Turn that judgment into a number with one S-shaped curve.

To teach by example, someone has to write the perfect answer.

That’s the catch. Writing one is hard — and even a careful answer can overstate, mislead, or be technically correct yet useless.

You don’t have to write it to know it’s worse

Read these two replies to “Is garlic a good natural antibiotic for colds?”

• “It can be helpful against colds, but may make you stink.”
• “It might be one of the best natural antibiotics out there, so it would help if you have a cold.”

You can’t easily write the ideal answer. But you know the second one is worse — it’s overconfident, a quiet flavour of telling you what sounds good over what’s true.

That’s the whole idea. People don’t need to know how to produce a good answer. They just need an opinion about which of two answers is better. Judging is far easier than writing.

So we collect preference data: a prompt, two sampled answers, and a verdict — a chosen (winner) and a rejected (loser). Written o_i > o_j for prompt x. This is the raw material for teaching the model what good looks like.

Today

The verdict need not come from a hired human. Reddit and Stack Overflow upvotes already rank answers for free. Or go fully synthetic — UltraFeedback prompts several LLMs for answers, then has GPT-4 rank them. Same shape: a winner and a loser.

Turn a verdict into a number

A verdict is just this one beats that one. To train on it we need a number.

So assume each answer has a hidden score z — how good it is. The better answer should usually win. But “usually” matters: if the two answers are close, the verdict could flip.

The trick is to look only at the difference z_i − z_j and squash it through the S-shaped logistic sigmoid:

P(o_i > o_j | x) = sigmoid(z_i − z_j) = 1 / (1 + e^−(z_i − z_j))

This is the Bradley–Terry model (1952). Read off the three cases:

• difference near 0 → probability near 0.5 — a toss-up, no real preference
• big positive difference → probability near 1 — o_i almost always wins
• big negative difference → probability near 0 — o_j almost always wins

Slide the two scores. The only thing that moves the curve is their difference — push it apart and the S saturates toward a sure winner; bring them level and it sits at a coin-flip 0.5.

Try it

set both scores equal and watch the probability lock onto 0.5. Now nudge one up a hair — the verdict barely leans. Nudge it far — it becomes near-certain.

The model only ever sees the gap, not the raw scores — which is why preferences map to it so cleanly. Shift both scores by the same amount and nothing changes.

Go deeper: it's just logistic regression in disguise

Start from what a probability implies, not from the sigmoid. The log-odds of an event is log( P / (1 − P) ). Set that equal to the score difference δ = z_i − z_j:

log( P / (1 − P) ) = δ

Exponentiate both sides, then solve for P:

P / (1 − P) = e^δ ⟹ P = e^δ / (1 + e^δ) = 1 / (1 + e^−δ) = sigmoid(δ)

So the sigmoid isn’t a magic curve — it’s the only shape consistent with “the score difference is the log-odds of preferring the winner.” That’s the same δ the logit plays in ordinary binary logistic regression, which is also why training on preferences reuses the familiar cross-entropy loss.

We now have a clean target: a probability we can fit. Next, we hand that hidden score over to a model and let it learn to be the judge.

Sources · 8

1. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd ed., draft of January 6, 2026). Stanford University. Ch. 9 (Post-training: Instruction Tuning, Alignment, and Test-Time Compute).
2. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862.
3. Bradley, R. A., & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 324-345.
4. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). arXiv:2203.02155.
5. Ethayarajh, K., Choi, Y., & Swayamdipta, S. (2022). Stanford Human Preferences Dataset (SHP).
6. Lambert, N., Tunstall, L., Rajani, N., & Thrush, T. (2023). HuggingFace H4 Stack Exchange Preference Dataset.
7. Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., & Sun, M. (2024). UltraFeedback: Boosting Language Models with Scaled AI Feedback. ICML 2024. arXiv:2310.01377.
8. Wang, Z., Dong, Y., Zeng, J., Adams, V., Sreedhar, M. N., Egert, D., Delalleau, O., Scowcroft, J., Kant, N., Swope, A., & Kuchaiev, O. (2024). HelpSteer: Multi-Attribute Helpfulness Dataset for SteerLM. NAACL HLT 2024. arXiv:2311.09528.

Full bibliography →