Evidence adds points

Something is behind a door. Cat or not-cat? Before any clue arrives, call it even: odds of 1:1.

Now a clue: it’s furry. Furry things are, say, four times likelier if it’s a cat. Your odds multiply by 4 → 4:1.

Another clue: it meows. Suppose this is ten times as likely for a cat as for a non-cat. If the clues are independent within cat cases and within non-cat cases, their evidence factors multiply: 40:1.

That assumption is conditional independence: after you are told whether it is a cat, learning one clue does not change the probability of another. Without it, the evidence supplied by a new clue may depend on the clues already seen.

Multiply on one ruler, add on the other

But chapter one told us what to do with anything that multiplies: look at it on the log ruler. There, each clue becomes a fixed number of points — furry is +1.4, meows is +2.3, barks is −3.0 — and evidence simply adds:

0 + 1.4 + 2.3 = +3.7 log-odds → sigmoid(+3.7) ≈ 97.6% cat

Belief-updating becomes bookkeeping. Every clue contributes its points, positive or negative; the running total is where you stand on the straight ruler; the sigmoid translates the total back into a probability whenever you want one.

Toggle clues on and off. The odds multiply, the points add, and the final total rides the sigmoid back to a probability.

Try it

turn on furry and meows, then add it barks. Watch the −3.0 points nearly cancel the evidence for cat — one strong counter-clue drags the whole total back toward unsure. The order you add clues never matters: addition doesn’t care.

The connection to logistic regression

Logistic regression uses a similar form: multiply each feature by a learned weight, add an intercept, and pass the total through a sigmoid. It fits an additive model of log-odds directly from labeled examples. It does not require the conditional-independence assumption used in our Bayesian example.

And it’s why the reward-model story in the preferences chapter reuses everything here: the Bradley–Terry gap z_A − z_B is a total of evidence on the log-odds ruler, and training it uses the same cross-entropy loss as logistic regression, because underneath they are the same object.

Go deeper: softmax is the many-door version

Two hypotheses need one number (the log-odds). What if there are 50,000 doors — say, every word in a vocabulary? Give each candidate its own score z_i on the log ruler, undo the log to get each one’s strength e^(z_i), and divide by the total so the strengths become shares:

P(i) = e^(z_i) / Σ_j e^(z_j)

That’s softmax — the sigmoid’s many-way generalization (the sigmoid is exactly softmax over two options with one score pinned at 0). It’s the machine at the mouth of every language model, turning scores back into words; temperature is nothing but stretching or shrinking the score ruler before the trip back.

Today

Real clues often overlap. Fitting logistic-regression weights can account for some of that overlap, but a linear score does not automatically capture every interaction. Sometimes the effect of one feature depends on another, and the model needs a corresponding interaction feature or a more flexible form.

Evidence moves beliefs one clue at a time. But a walk is more than its steps — it matters where you started. Next: the number almost everyone forgets, and the famous “paradox” that dissolves the moment you remember it.

Sources · 4

1. Jaynes, E. T. (2003). Probability Theory: The Logic of Science. Cambridge University Press. Ch. 4 (Elementary Hypothesis Testing — evidence in decibels).

2. Berkson, J. (1944). Application of the Logistic Function to Bio-Assay. Journal of the American Statistical Association, 39(227), 357–365. (Coins the term “logit”.)

3. Bradley, R. A., & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 324–345.

4. Scikit-learn developers. Naive Bayes: Bayes’ rule and the conditional-independence factorization.

Full bibliography →