ASR and TTS evaluation
Compare a machine transcript with the human answer, then use listeners to judge generated speech.
An automatic speech recognition (ASR) system turns speech into text. To measure it, call the human transcript the reference and the system’s transcript the hypothesis.
Word error rate
Section titled “Word error rate”Minimum edit distance asks for the fewest word changes needed to make the hypothesis match the reference. There are three possible errors:
- Substitution (S): replace a wrong word.
- Insertion (I): remove an extra word added by the system.
- Deletion (D): restore a reference word the system missed.
Word error rate (WER) compares those errors with N, the number of reference words:
WER = 100 × (S + D + I) ÷ N
For the slide example:
- Reference: “the cat sat on the mat”
- Hypothesis: “a cat sat on that mat”
the → a and the → that are substitutions. Thus S = 2, I = 0, D = 0, and N = 6, so WER = 33.3%.
Normalize before comparing
Section titled “Normalize before comparing”Text normalization applies the same writing rules to both transcripts. Lowercase them, remove punctuation, expand numbers, and handle contractions consistently. Otherwise, “3 cats” and “three cats” would produce a 50% WER even though they say the same thing.
WER is useful, but limited:
- Every error has the same cost.
- Synonyms such as “automobile” and “car” count as different.
- Many insertions can push WER above 100%.
- WER counts edits, not meaning.
When comparing two systems on the same utterances, the MAPSSWE test compares their utterance-level errors and produces a p-value. A p-value is a number used to judge whether the difference could be random. In the slides, p < 0.05 means the difference is statistically significant.
Evaluate generated speech
Section titled “Evaluate generated speech”Mean opinion score (MOS) is the average of human ratings from 1 to 5: 1 means bad or unintelligible; 5 means excellent and human-like.
Comparative MOS (CMOS) is a direct A/B comparison. Negative scores mean System A is worse, 0 means no preference, and positive scores mean System A is better.
Automatic proxy metrics add useful clues: ASR WER tests intelligibility, speaker verification tests speaker similarity, and phoneme accuracy tests pronunciation. They supplement human MOS; they do not replace it.
Key takeaways
Section titled “Key takeaways”- WER counts the fewest word edits relative to reference length.
- Use identical text normalization before scoring.
- WER is standard, but it does not measure meaning.
- Use statistical significance when comparing systems.
- Use MOS or CMOS as the main TTS evaluation.