Audio AI Foundations β What Demucs, Mel-Band RoFormer, and GPT-SoVITS Share
STFT spectrograms, Transformer attention, encoder-decoder, PyTorch β the shared DNA of audio AI
Analyzing Demucs, Mel-Band RoFormer Karaoke, and GPT-SoVITS revealed shared patterns. Different purposes, same skeleton.
Foundation 1: STFT β Turning Sound into Pictures
STFT (Short-Time Fourier Transform) converts audio waveforms into time Γ frequency 2D matrices. This is where all audio AI starts.
Waveforms only have a time axis. STFTs produce spectrograms β time Γ frequency Γ intensity. Now you can see "vocals around 440Hz, bass below 100Hz."
spec = torch.stft(wav, n_fft=2048, hop_length=441, return_complex=True)
All three models use this transform.
Foundation 2: Transformer β Capturing Long-Range Relationships
All three use Self-Attention at their core. CNN receptive fields are limited by kernel size. Audio needs:
Rhythm patterns spanning seconds
Harmonics at 2Γ, 3Γ, 4Γ the fundamental frequency
Chorus repeating 30 seconds later
Self-Attention computes relationships between all positions directly. No distance limit.
Foundation 3: Encoder-Decoder β Decompose and Reconstruct
All three compress input (encode), process it, then restore original form (decode). Same pattern: high-dimensional input β low-dimensional latent space β meaningful processing β original dimension restoration.
Foundation 4: PyTorch + GPU + Pre-trained Models
All three: PyTorch implementation, CUDA/MPS GPU acceleration, pre-trained checkpoints from HuggingFace/GitHub. pip install + model download + 3 lines of code.
Foundation 5: Audio Decomposition
Different directions, same essence β decomposing complex audio signals into meaningful components.
Foundation 6: Masking vs Generation
Separation models (Demucs, RoFormer) use masking: original spectrogram Γ mask = separated source. Synthesis models (GPT-SoVITS) generate new audio tokens. But both work in the frequency domain.
Why This Matters
Four questions classify any audio AI model:
1. How is input represented? (STFT? mel? raw?)
2. Where is attention applied? (time? frequency? both?)
3. Encoder-decoder structure? (what's compressed, what's restored?)
4. Masking or generation? (separation or synthesis?)
Key Concepts
STFT converts audio waveform to time Γ frequency spectrogram β the starting point of audio AI
Transformer Self-Attention captures long-range time and frequency dependencies
Encoder-decoder compresses high-dimensional input to latent space β process β reconstruct
Separation models use masking (originalΓmask), synthesis models generate (new token output)