CNNs for speech

Speech has local structure: a sound at one moment is closely related to nearby moments. The same sound can also occur anywhere in a recording. A useful model should therefore reuse one small detector across the whole timeline.

A convolutional neural network (CNN) does exactly that. It slides learned detectors across the input. Each detector can discover a useful acoustic pattern, such as a short burst or a change in frequency. These are learnable filters because the model finds their numbers from training examples.

Slide a small filter through time

The detector is a kernel, a small group of learned weights. A weight is a number that controls how strongly one input value affects the result.

A width-1 kernel reads one time position at a time. It transforms each position independently, so it is also called a pointwise transformation.

A width-3 kernel reads three neighboring positions. It multiplies them by three learned weights and adds the results. The same kernel then moves to the next local window. With stride 1, shifting a pattern shifts its response along the output, apart from edge effects. This is translation equivariance: the detector follows the pattern in time, rather than ignoring where it occurred.

A channel is one track of output values. With multiple channels, many kernels learn different acoustic patterns at the same time. A kernel can also span all input channels—for example, many Mel-frequency bands—while looking across a few time positions.

Control the length and context

Padding means adding zeros at the sequence edges. Without padding, a width-3 kernel cannot be centered on the first or last position, so the output becomes shorter. Padding by one position on each side can keep the input and output lengths equal.

A stride is how far the kernel moves before producing its next output. Stride 1 moves one position. Stride 2 skips every other position and produces about half as many outputs. This is efficient downsampling: reducing sequence length so later layers have less work.

Whisper uses two convolution layers: the first has stride 1 and the second has stride 2. Together they reduce 3,000 frames to 1,500. wav2vec 2.0 uses seven convolution layers and reduces 16 kHz waveform input to about 50 frames per second, roughly a 320-fold reduction.

The receptive field is the part of the original input that can affect one output position. One width-3 layer sees three positions. Two stacked width-3 layers see five. Deeper stacks see more context, but making the field too broad can reduce temporal precision, meaning exact timing becomes less clear.

Combine local and long-range processing

Convolutions are efficient because they inspect a small local window. For T time positions and kernel width k, their work grows roughly as T × k. Self-attention, which lets each position compare itself with every other position, grows roughly as T².

The slides therefore use a convolutional front end + transformer. The CNN first learns local patterns and shortens the audio. A transformer then connects information across the remaining sequence to build wider context.

Key takeaways

1D convolutions reuse learned local filters across time.
Multiple channels capture different acoustic patterns.
Padding handles edges; stride controls downsampling.
The receptive field grows as convolution layers are stacked.
A convolutional front end makes later transformer processing more efficient.

Sources · 2

Full bibliography →