Audio Tokenization (Neural Audio Codec)
Converting audio into discrete tokens for LLM-like processing
Audio Tokenization is a core technology changing the paradigm of speech/audio AI. Previously, TTS used a continuous pipeline of textβmel spectrogramβwaveform, but Neural Audio Codec converts audio into "tokens." Meta's EnCodec and Google's SoundStream are representative examples. Audio is compressed with an encoder and then converted to multi-layer discrete codebook indices via Residual Vector Quantization (RVQ). The first codebook handles semantic/prosody information, while subsequent codebooks handle acoustic details. By generating these token sequences with a Transformer, high-quality audio can be produced in the same way as text generation. VALL-E, Bark, GPT-SoVITS, etc. utilize this paradigm.
Key Concepts
Compress audio waveform into latent representation using CNN encoder
Residual Vector Quantization (RVQ): quantize latent representation into multi-layer codebooks
Layer 1 codebook: semantic/prosody information (semantic tokens) β most important information
Layer 2-8 codebooks: acoustic details (acoustic tokens) β progressively adding detail
CNN decoder reconstructs audio waveform from quantized codes
Result: audio represented as discrete token sequences like text β can be input to Transformers
Pros
- ✓ Audio can be processed by Transformers like text
- ✓ LLM advances directly applicable to speech/audio
- ✓ Very high compression rate (24kHz β 75 tokens per second)
- ✓ Foundation for multimodal unified models (text+audio in same token space)
Cons
- ✗ Audio quality loss during quantization
- ✗ Codebook size/layer count tuning required
- ✗ Difficulty maintaining consistency in long audio
- ✗ Large training data required (codec training itself)