principles.fyi · the brain · concept

weight tying

Some models reuse one word-list both to read and to guess.

word_scores = hidden_vector · (embedding_table)ᵀ

When the model reads, it looks each word up in a big table that turns words into number-lists. When it guesses the next word, it needs the opposite: turn its number-list back into scores for every word. Weight tying says "use the same table for both jobs, just flipped around" instead of learning two separate ones. That means fewer numbers to store, and the word's "spelling-in" matches its "spelling-out," so reading and guessing stay friends.