Demucs Code Analysis โ How One Song Gets Split into Vocals, Drums, Bass, and Guitar
Meta/Facebook Research's Hybrid Transformer source separation model โ architecture, inference pipeline, real-world usage
What Is Demucs?
Demucs (Deep Extractor for Music Sources) is a music source separation model by Meta's Facebook AI Research (FAIR). First released in 2019 (v1), now at v4 (htdemucs).
Input: one complete song. Output: 4 stems:
vocals โ all vocals (lead + backing)
drums โ drums/percussion
bass โ bass
other โ everything else (guitar, piano, synths, etc.)
from demucs.pretrained import get_model
from demucs.apply import apply_model
model = get_model("htdemucs")
sources = apply_model(model, wav[None], device="cuda")[0]
# sources.shape = [4, 2, samples]
Version History
| Version | Name | Architecture | Year |
|---|---|---|---|
| v1 | demucs | Time-domain U-Net | 2019 |
| v2 | demucs | Improved time-domain | 2021 |
| v3 | mdx/hybrid | Time+frequency hybrid | 2021 |
| v4 | htdemucs | Hybrid Transformer | 2022 |
v1-v2 were time-domain CNN-based. v3 added STFT (frequency domain) for hybrid approach. v4 introduced Transformer attention.
htdemucs Architecture
Two parallel paths:
1. Time-Domain Path (Temporal Encoder-Decoder)
Processes raw waveform directly
1D CNN encoder โ Bottleneck โ 1D CNN decoder with skip connections
Captures fine temporal details (transients, attacks)
2. Frequency-Domain Path (Spectral Encoder-Decoder)
Processes STFT spectrogram
2D CNN encoder โ Bottleneck โ 2D CNN decoder
Captures frequency patterns (harmonics, pitch)
3. Cross-Domain Transformer (Core Innovation)
Both encoder outputs feed into Transformer
Self-attention + cross-attention exchanges information between domains
This is v4's key improvement โ actually combining both domains' strengths
Final output sums both decoders' results.
Inference Pipeline Detail
demucs.apply.apply_model() flow:
1. Resample to 44,100Hz stereo
2. Split into chunks (~10s, 25% overlap)
3. Per chunk:
a. With shifts=1: process original + random time-shifted version (TTA)
b. Time path: waveform โ 1D CNN โ Transformer โ 1D CNN
c. Freq path: STFT โ 2D CNN โ Transformer โ 2D CNN โ ISTFT
d. Sum both path outputs
4. Blend overlapping regions with fade-in/fade-out
5. Return 4-stem tensor: [4, 2, samples]
In Nightingale: extracts vocals only, computes original - vocals = instrumental.
UVR Karaoke vs Demucs
| UVR Mel-Band RoFormer Karaoke | Demucs htdemucs | |
|---|---|---|
| Stems | 2 (lead vocals / karaoke) | 4 (vocals / drums / bass / other) |
| Vocal definition | Lead vocals only | All vocals (lead + backing) |
| Backing vocals in instrumental | Yes (by design) | No (classified as vocals) |
| Model size | 913 MB | ~80 MB |
| Vocal SDR | 10.20 dB (lead only) | ~9.20 dB (all vocals) |
| Karaoke suitability | High | Medium |
UVR Karaoke is better for karaoke (backing vocals remain). Demucs is better for remixing, sampling, and instrument practice.
Practical Use Cases
- Karaoke apps โ vocal removal for accompaniment playback
- Music remixing โ manipulate individual stems
- Instrument practice โ remove one instrument to practice its part
- Music analysis/education โ decompose song structure by parts
- Mastering โ adjust individual elements in pre-mixed tracks
- Copyright detection โ isolate vocals for melody/lyrics comparison
Limitations
Perfect separation impossible โ "bleeding" between stems
CPU inference takes 2-5x song length. GPU recommended
Cannot distinguish backing from lead vocals
Quality drops on electronic music with blurry instrument boundaries
Key Concepts
Resample input audio to 44,100Hz stereo
Split into ~10s chunks (25% overlap)
Time path: waveform โ 1D CNN encoder โ Cross-Domain Transformer โ 1D CNN decoder
Freq path: STFT โ 2D CNN encoder โ Cross-Domain Transformer โ 2D CNN decoder โ ISTFT
Sum both path outputs โ return 4 stems (vocals/drums/bass/other)