Evolution of Speech Models
Rule-based → Deep learning pipeline → Token-based → Unified multimodal
Speech synthesis technology has gone through 5 major transitions.
1st Gen (1960s-1990s) — Rule-based/Concatenative Synthesis: Recording phoneme-level speech segments and concatenating them, or generating via formant synthesis rules. Characterized by robotic-sounding speech.
2nd Gen (2000s-2015) — Statistical Parametric: HMM (Hidden Markov Model) based. Learning statistical models of speech and generating from parameters. Improved naturalness but still mechanical.
3rd Gen (2016-2022) — Deep Learning Pipeline: Tacotron/FastSpeech generates text→mel spectrograms, WaveNet/HiFi-GAN converts to waveforms as vocoders. Achieved near-human naturalness. But requires multi-stage pipeline: text encoder→mel decoder→vocoder.
4th Gen (2023-Present) — Token-based LLM Approach: Neural Audio Codec (EnCodec, SoundStream) converts audio to discrete tokens. Transformer generates text tokens→audio tokens. VALL-E, Bark, GPT-SoVITS are representative. Voice cloning from just 3 seconds (Zero-shot), natural emotion/intonation delivery.
5th Gen (Future) — Unified Multimodal Models: Processing text, speech, and images in a single token space like GPT-4o. Natural conversation end-to-end without separate TTS modules. Real-time emotion understanding and speech expression integrated.
Key Concepts
1st Gen: find and concatenate segments from phoneme DB (Concatenative) — monotonous and unnatural
2nd Gen: model speech parameters with HMM → synthesize with vocoder — naturalness up, but mechanical
3rd Gen: Tacotron/FastSpeech (text→mel) + WaveNet/HiFi-GAN (mel→waveform) — human-level naturalness
4th Gen: tokenization via Neural Audio Codec → Transformer generates tokens — Zero-shot voice cloning possible
5th Gen (Future): unified processing of text+speech+images in one token space — true multimodal conversational AI
Pros
- ✓ Dramatic improvement in naturalness and expressiveness each generation
- ✓ Directly applying LLM technology to speech via token-based approach
- ✓ Reduced data collection burden with Zero-shot voice cloning
- ✓ Unified multimodal is the key to real-time conversational AI
Cons
- ✗ Rapidly increasing risk of deepfake/voice phishing abuse
- ✗ Token-based models require large amounts of training data
- ✗ Latency challenges in real-time streaming
- ✗ Ethical issues of voice cloning without speaker consent