TTS and VALL-E
See how VALL-E turns text and a short voice sample into personalized speech.
Text-to-speech (TTS), also called speech synthesis, turns written text into spoken audio.
- Input: text plus a voice sample.
- Output: a speech waveform in that voice.
A waveform is the ordered list of sound measurements played by a speaker. TTS is a one-to-many mapping: the same sentence can be spoken by different voices, at different speeds, and with different emotions. Modern TTS uses a reference recording to show the system how the requested voice should sound.
Zero-shot TTS
Section titled “Zero-shot TTS”Traditional TTS often trained one model for one speaker. Zero-shot TTS can generate speech for a voice not seen during training. The slides use VALL-E as the example: it was trained on 60K hours from more than 7,000 speakers and uses a three-second voice sample.
“Zero-shot” does not mean “without an example.” The text supplies the content, while the short recording supplies the voice.
VALL-E predicts compact sound codes
Section titled “VALL-E predicts compact sound codes”A text tokenizer turns the requested words into numbered text pieces. An audio codec compresses the voice sample into numbered sound choices called codec tokens. A codebook is the fixed list of sound choices those numbers refer to.
VALL-E then uses two stages:
- Codebook 1 carries the most important information for intelligibility and voice identity. An autoregressive (AR) Transformer generates it sequentially: each new code can use earlier codes.
- Codebooks 2–8 add acoustic detail. A non-autoregressive (NAR) Transformer generates their positions in parallel, making this refinement faster.
This creates a trade-off: use careful sequential prediction for the main structure and faster parallel prediction for the remaining detail. Finally, the codec decoder turns all generated codes into personalized speech.
Voice cloning and responsibility
Section titled “Voice cloning and responsibility”During training, the model hears the same content in different voices and different content in the same voice. It can therefore learn that content and voice identity are separate. At inference—the time when a trained model makes new output—the three-second sample supplies speaker characteristics and acoustic style. The generated speech matches that voice while saying new words.
This voice cloning ability creates three risks named in the slides:
- fraud and impersonation;
- non-consensual voice cloning;
- misleading audio content.
The slides also name three mitigations:
- audio watermarking, which adds a detectable marker;
- detection tools for identifying generated audio;
- usage policies that limit harmful use.
Key takeaways
Section titled “Key takeaways”- TTS turns text plus a voice sample into speech.
- VALL-E predicts codec tokens rather than raw sound measurements.
- AR generates codebook 1; NAR refines codebooks 2–8.
- A short sample can support zero-shot voice cloning.
- Voice cloning requires consent and responsible safeguards.