Encoder-decoder ASR with Whisper
Follow the encoder, cross-attention, and decoder as Whisper writes a transcript one token at a time.
Speech contains many more time steps than text. One second may become 50–100 frames, or short slices of sound, while the transcript may contain only a few tokens, such as letters or word pieces.
An encoder–decoder speech recognizer solves that mismatch with one listener and one writer:
waveform → log-Mel features → subsampling → encoder → decoder → text
A log-Mel feature is a compact description of the frequencies in one sound frame. Subsampling shortens the frame sequence so the model has fewer steps to process.
Encode the whole recording
Section titled “Encode the whole recording”The encoder turns the shortened sound sequence into contextual representations. A representation is a list of learned numbers. Contextual means that each list describes one moment using clues from the surrounding moments too.
Whisper uses Transformer encoder blocks. A Transformer is a model built from attention and per-position transformations. Self-attention lets sound positions compare with other sound positions, helping a weak clue become clearer from its context.
Look back while writing
Section titled “Look back while writing”The decoder predicts the transcript one token at a time. It needs the text written so far and the encoder’s sound information.
Cross-attention connects those two sequences. The decoder provides a query (Q)—what it needs now. The encoder provides keys and values (K, V)—where sound clues are located and what information they contain. Larger attention weights mean “look here more.” This lets the decoder learn an alignment, or connection between a text token and the sound frames that support it.
Each decoder block has three main sublayers:
- Causal self-attention reads earlier output tokens but cannot see future ones.
- Cross-attention reads the encoder’s complete sound sequence.
- A feedforward network transforms each position separately.
Each sublayer uses LayerNorm, which keeps its values well-scaled, and a residual connection, which carries the earlier representation around the sublayer.
Train with answers; infer without them
Section titled “Train with answers; infer without them”During training, teacher forcing gives the decoder the correct previous token. The cross-entropy loss becomes large when the correct next token receives low probability, pushing the model toward better predictions.
Whisper begins with special tokens that state the job: <|startoftranscript|> begins, <|en|> selects English, <|transcribe|> selects transcription, and <|notimestamps|> disables timestamps. <|endoftext|> ends generation.
For a new recording, no correct earlier tokens are available. Autoregressive generation predicts one token, adds it to the history, and repeats. Greedy search keeps the best immediate token. Beam search keeps several promising transcripts until an end token appears.
Key takeaways
Section titled “Key takeaways”- The encoder represents the audio; the decoder writes the text.
- Cross-attention lets each output token find useful sound evidence.
- Decoder blocks use causal self-attention, cross-attention, and a feedforward network.
- Teacher forcing trains the model; special tokens and search control inference.