ASR and audio representation

Start with ASR, TTS, waveforms, spectrograms, Mel features, and the basic challenges of recognizing speech.

Speech processing means using computers to work with spoken language. Its central problem is simple to state: speech is continuous, while words are discrete. Continuous sound changes smoothly over time. Discrete text is made from separate choices such as letters, word pieces, and words.

A microphone measures changing air pressure. Each measurement is a sample, and the ordered list of samples is a waveform. At a sampling rate of 16 kilohertz, written as 16 kHz, the microphone records 16,000 samples each second. That is a very long input for only a few spoken words.

The two main tasks

Automatic speech recognition (ASR) turns a waveform into a text transcription. A transcription is the written record of what was said.

waveform → text

Text-to-speech (TTS) travels in the other direction. It turns text into a new speech waveform.

text → waveform

Both tasks use sequence modeling: learning how an ordered list of inputs relates to an ordered list of outputs. ASR must reduce thousands of sound measurements to a few text tokens. A token is one output symbol, such as a character or a reusable piece of a word.

Why ASR difficulty changes

ASR is not one fixed test. Its difficulty varies along five dimensions, meaning five ways the task can change:

Vocabulary: A small-vocabulary system may recognize only a wake word such as “Alexa” or about 100 commands. An open-vocabulary system must handle arbitrary words and names. It can build unfamiliar words from characters or BPE subwords, common word chunks chosen automatically from training text.
Speakers: A speaker-dependent system is trained for one person. A speaker-independent system must work across differences in pitch, accent, and speaking rate.
Style: Read speech is usually clear and measured. Conversational speech contains interruptions, repairs, “um,” false starts, and overlapping voices.
Environment: Studio audio is cleaner than speech recorded in noisy rooms or from distant microphones.
Language: A single-language system solves a narrower problem than a multilingual system.

Researchers use a corpus, an organized collection of recordings and transcripts, to test a particular challenge. LibriSpeech contains clean read audiobooks. Switchboard contains spontaneous telephone conversations. CHiME tests speech in background noise. AMI contains multi-speaker meeting recordings, often captured from a distance. CORAAL represents African American English and helps test dialect coverage.

ASR remains difficult even within one corpus. Coarticulation means neighboring speech sounds affect one another. Speakers produce different acoustic signals, noise can cover useful evidence, and speaking rate changes the timing. An encoder reads the audio and builds context-aware number lists; a decoder uses those lists to produce text one token at a time. “Context-aware” means a sound is interpreted using nearby sound.

Represent sound at a useful rate

The raw waveform is the time-domain representation: sound shown as amplitude, or pressure strength, over time. It is accurate but too detailed. A phoneme—a sound category that can distinguish words—often lasts about 50–100 milliseconds, so a model does not need 16,000 decisions each second.

Speech also depends on frequency, how quickly a vibration repeats. The Short-Time Fourier Transform (STFT) examines 25-millisecond windows of audio, moving forward 10 milliseconds at a time. It produces a spectrogram, a grid showing frequency strength over time. This changes 16,000 samples per second into about 100 frames per second. A frame is one short time slice.

Human hearing does not separate all frequencies equally. The Mel scale uses narrower bands at low frequencies and broader bands at high frequencies. A common Mel spectrogram has 80 bands, so one second becomes roughly 100 frames by 80 values.

Traditional systems often use Mel-Frequency Cepstral Coefficients (MFCCs). They compress each Mel frame to 13 coefficients, then add deltas and delta-deltas, which describe first and second changes over time, for 39 features per frame.

Modern systems take two main approaches. wav2vec 2.0 and HuBERT use convolutional neural networks (CNNs) to learn features from the raw waveform. Whisper uses an 80-dimensional log-Mel spectrogram, where “log” compresses a wide range of sound strengths into a smaller numeric range. Either way, strided convolutions perform downsampling, meaning they keep fewer time positions. This helps solve the rate mismatch between many sound frames and few text tokens.

Key takeaways

ASR maps a waveform to text; TTS maps text to a waveform.
ASR difficulty changes with vocabulary, speakers, style, environment, and language.
STFT turns short waveform windows into a time-by-frequency spectrogram.
Mel bands and MFCCs give compact descriptions of frequency content.
Strided CNNs shorten the sound sequence before later processing.

Sources · 2

Full bibliography →