Text-to-Speech (TTS)
Converting text to natural-sounding speech
Modern TTS has evolved from rule-based/concatenative synthesis to deep learning. Models like Tacotron/FastSpeech convert text to mel spectrograms, and vocoders (WaveNet, HiFi-GAN) convert these to waveforms. The latest approach is LLM-based TTS (VALL-E, Bark), going text โ audio tokens โ waveforms. Zero-shot Voice Cloning is now possible from just 3 seconds of reference speech, and OpenAI's GPT-4o handles text understanding + speech generation in a single model.
Key Concepts
Text preprocessing: normalization, phoneme conversion, prosody analysis
Text encoder: convert phoneme sequence to latent representation
Decoder: latent representation โ generate mel spectrogram (time-frequency representation)
Vocoder (HiFi-GAN, WaveNet): mel spectrogram โ audio waveform conversion
Latest LLM-based: text โ audio tokens (Neural Codec) โ waveform
Post-processing: noise removal, volume normalization
Pros
- ✓ Natural speech indistinguishable from humans
- ✓ Zero-shot voice cloning
- ✓ Controllable emotion/intonation/speed
- ✓ Multilingual support
Cons
- ✗ Risk of deepfake/fraud misuse
- ✗ Emotional expression still imperfect in some cases
- ✗ Stability degradation on long texts
- ✗ Ethical issues of voice cloning without speaker consent