Self-supervised speech models with HuBERT
See how HuBERT masks audio, predicts machine-made sound groups, improves those groups, and then learns ASR.
Accurate transcripts are expensive to create, but unlabeled recordings—audio with no written answer—are plentiful. Self-supervised learning uses the recording itself to make a practice task. The model can first learn how speech behaves, then use a much smaller labeled set to learn text.
The HuBERT practice game
Section titled “The HuBERT practice game”HuBERT follows one simple flow:
cluster → mask → predict → recluster → fine-tune
A frame is a short slice of audio. HuBERT needs one discrete answer for each frame, but raw audio has no token labels. It creates them by grouping similar frames.
1. Create the first targets
Section titled “1. Create the first targets”An MFCC is a compact, traditional description of one speech frame. K-means groups similar MFCC vectors. A vector is a list of numbers; a cluster is one group; its cluster label is the group’s ID.
In the slides’ first stage, HuBERT clusters MFCCs from the entire training set into 100 groups. These IDs are training targets, not human phoneme names.
2. Hide frames and predict their labels
Section titled “2. Hide frames and predict their labels”A CNN is a stack of sliding sound-pattern detectors. Its encoder changes the waveform into frame embeddings, or learned number-lists for short sound moments. Masking hides some neighboring embeddings. A Transformer uses attention to combine information across time and predict the hidden frames’ cluster labels.
The model cannot simply copy the hidden sound. To guess correctly, it must learn patterns across the recording.
3. Build better targets
Section titled “3. Build better targets”MFCC clusters are only a starting point. After the first training stage, HuBERT’s learned representations contain stronger speech clues. The slides take the output of layer 6 from 10% of the training data, run k-means again, and create 500 new groups.
The model then repeats the masked prediction task with these improved targets. This is the slide’s two-stage target creation: rough clusters first, learned-representation clusters second.
4. Fine-tune for recognition
Section titled “4. Fine-tune for recognition”Self-supervised practice teaches sound structure, not written words. Fine-tuning continues training on labeled audio for the final task. The slides add a CTC head, an output layer that maps sound frames to text without requiring a timestamp for every letter. The seven CNN layers are shown frozen, meaning their learned numbers are not changed during this step.
Why it helps
Section titled “Why it helps”Predicting masked clusters encourages the model to learn phonetic structure—which speech sounds fit together; speaker normalization—the same sound across different voices; and coarticulation—how neighboring sounds blend.
Traditional supervised systems may need thousands of labeled hours. The slides show self-supervised pretraining followed by only 10–100 labeled hours. That makes low-resource ASR, speech recognition with little transcribed data, much more practical.
Key takeaways
Section titled “Key takeaways”- HuBERT learns first from unlabeled audio.
- It masks frames and predicts k-means cluster labels.
- A second clustering stage creates better targets from learned representations.
- Fine-tuning with a CTC head turns the learned sound knowledge into ASR.