principles.fyi · the brain · concept

encoder-decoder

Two transformers in a row: one reads a whole input, the other writes a fresh output conditioned on it.

encoder(input) -> context; decoder(context) -> output sequence

An encoder-decoder pairs the two shapes. The encoder reads the entire input sentence bidirectionally and turns it into a set of context vectors; the decoder then writes a new sequence left-to-right, attending back to those vectors at every step (cross-attention). It shines when the output is a transformation of a specific input — translation (English to French), summarization, or speech-to-text — because the decoder is anchored to a fixed, fully-read source rather than continuing free-form text. T5 is the classic example.