GPT-SoVITS Practical Analysis
How Pre-training and Fine-tuning work in actual code
GPT-SoVITS is an open-source Few-shot TTS system that can clone a speaker's voice with just 1 minute of audio. It consists of a 2-stage pipeline.
Stage 1 β GPT (Text-to-Semantic): Takes text and reference audio as input to generate Semantic Tokens. A 24-layer Transformer that predicts tokens one by one in an autoregressive manner. The key is using semantic tokens extracted by CN-HuBERT (a self-supervised learning SSL model) as prediction targets.
Stage 2 β SoVITS (Semantic-to-Audio): Converts Semantic Tokens into actual audio waveforms. Based on VITS architecture, consisting of VQ (Vector Quantization) codebook + Flow model + HiFi-GAN decoder.
What is learned in Pre-training:
CN-HuBERT: Learning semantic speech representations from 2,000-5,000 hours of multi-speaker data (SSL)
Chinese-RoBERTa: BERT feature extractor for text (separately pre-trained)
SoVITS VQ codebook: Learning universal speech codebook from multi-speaker data
GPT model: Learning textβsemantic token mapping from multi-speaker data
What is done in Fine-tuning:
User provides 1 minute of audio + text
Data preprocessing: phoneme extraction, BERT/SSL/Semantic token extraction
s1_train.py: Fine-tune GPT model for 20 epochs, lr=0.01
s2_train.py: Fine-tune SoVITS model for ~100 epochs, lr=0.0001
In v3, LoRA (rank=32) is applied to the CFM module for efficient fine-tuning
Key Concepts
Pre-training 1: CN-HuBERT: self-supervised learning on thousands of hours of speech data β learn semantic speech representations
Pre-training 2: SoVITS VQ: learn universal speech codebook from multi-speaker data β semantic-acoustic mapping
Pre-training 3: GPT model: learn textβsemantic token autoregressive generation from multi-speaker data
Fine-tuning data preparation: 1 min audio β phoneme, BERT features, SSL features, semantic token extraction
Fine-tuning 1: s1_train.py: train GPT model on target speaker data for 20 epochs (lr=0.01)
Fine-tuning 2: s2_train.py: train SoVITS model on target speaker data for ~100 epochs (lr=0.0001)
Inference: text β GPT generates semantic tokens β SoVITS converts to target speaker audio waveform
Pros
- ✓ High-quality voice cloning with just 1 minute of data
- ✓ Open source β code analysis and customization possible
- ✓ 2-stage pipeline allows independent optimization of each stage
- ✓ LoRA support in v3 β improved fine-tuning efficiency
- ✓ Practical understanding of Pre-training/Fine-tuning concepts
Cons
- ✗ Depends on pre-trained model quality
- ✗ Chinese-centric pre-training β possible performance degradation in other languages
- ✗ GPU required (VRAM 6GB+ for fine-tuning)
- ✗ Risk of voice deepfake abuse