principles.fyi · the brain · concept

bidirectional

Each word gets to read both the words before it AND the words after it.

context(word) = words on the left + words on the right (no causal mask)

Bidirectional means every token's understanding is built from context on both sides at once — left and right — instead of only what came before. Book 01's writer used the causal mask to hide the future so it could predict the next word; an understander drops that mask entirely, so attention is free to look in both directions. That two-sided view is exactly why these models are so good at grasping what a word means in context, but it's also why they can't generate text left-to-right the way the writer does.