Audio AI Hello World — 7 Projects in 5 Minutes with Pre-trained Models

All runnable code. Python 3.10+ required.

0. From Scratch — The PyTorch Pipeline Behind Every Audio AI Model

Calling CLI tools is "using," not "understanding." Here's what actually happens inside Demucs and RoFormer, in PyTorch:

Core flow:

raw audio → STFT → spectrogram → neural net (mask prediction) → original × mask → ISTFT → separated audio

import torch, torchaudio

# 1. Load audio
wav, sr = torchaudio.load("song.mp3")  # [2, samples]

# 2. STFT → frequency domain
spec = torch.stft(wav, n_fft=2048, hop_length=441,
                  window=torch.hann_window(2048), return_complex=True)
magnitude, phase = spec.abs(), spec.angle()

# 3. Neural net predicts mask (simplified — real models use Transformers)
mask = model(magnitude)  # → [2, 1025, time] values 0~1

# 4. Apply mask
vocal_spec = (magnitude * mask) * torch.exp(1j * phase)

# 5. ISTFT → back to audio
vocals = torch.istft(vocal_spec, n_fft=2048, hop_length=441,
                      window=torch.hann_window(2048), length=wav.shape[1])
torchaudio.save("vocals.wav", vocals, sr)

The mask is the key: values near 1.0 = "this time-frequency bin is vocals." Demucs (26.3M params) and RoFormer (84.2M params) are just more sophisticated versions of this mask predictor.

1. Source Separation — Remove Vocals from Songs

Model: Demucs htdemucs (Meta, ~80MB)
Expected: Input MP3 → outputs vocals.wav + no_vocals.wav

pip install demucs torchcodec
demucs song.mp3 --two-stems vocals

2. Speech Recognition (STT) — Audio to Text

Model: OpenAI Whisper (tiny=75MB to large-v3=3GB)

import whisper
model = whisper.load_model("tiny")
result = model.transcribe("audio.mp3")
print(result["text"])

3. Text-to-Speech (TTS)

Model: Coqui TTS XTTS-v2 (~1.8GB)

from TTS.api import TTS
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2")
tts.tts_to_file(text="Hello world", language="en", file_path="output.wav")

4. Pitch Detection

import librosa
y, sr = librosa.load("vocal.wav")
f0, _, _ = librosa.pyin(y, fmin=80, fmax=1000, sr=sr)

5-6. Audio Classification / Music Tagging

Model: AST (MIT, ~350MB)

from transformers import pipeline
classifier = pipeline("audio-classification", model="MIT/ast-finetuned-audioset-10-10-0.4593")
print(classifier("audio.wav")[:3])

7. Music Generation

Model: MusicGen (Meta, 300MB-3.3GB)

from transformers import pipeline
generator = pipeline("text-to-audio", model="facebook/musicgen-small")
music = generator("upbeat electronic dance music")

Pre-trained Model Directory

All run on CPU without GPU. Slower, but results come out.