Speech Processing

A concise, slide-based path through ASR, audio features, CNNs, CTC, codecs, TTS, and evaluation.

Contents

1. ASR and audio representation — Start with ASR, TTS, waveforms, spectrograms, Mel features, and the basic challenges of recognizing speech.
2. CNNs for speech — See how convolutional filters find local sound patterns, shorten the audio sequence, and prepare it for a transformer.
3. Audio tokenization and codecs — Compress a waveform into audio tokens with an encoder, vector quantizer, residual codebooks, and decoder.
4. Encoder-decoder ASR with Whisper — Follow the encoder, cross-attention, and decoder as Whisper writes a transcript one token at a time.
5. Self-supervised speech models with HuBERT — See how HuBERT masks audio, predicts machine-made sound groups, improves those groups, and then learns ASR.
6. CTC: alignment without timestamps — Use CTC blanks and collapsing rules to learn the word “dinner” without labeling when each letter was spoken.
7. TTS and VALL-E — See how VALL-E turns text and a short voice sample into personalized speech.
8. ASR and TTS evaluation — Compare a machine transcript with the human answer, then use listeners to judge generated speech.

Sources and further reading are in the bibliography.