๐Ÿ“Š

AI Preprocessing Difficulty โ€” Text, Image, Audio, Video, 3D in Order of Complexity

model(input) is one line, but the preprocessing to create that input is completely different per domain

All AI inference follows this structure:

processed_input = preprocess(raw_data)   # โ† differs by domain
output = model(processed_input)          # โ† always the same
result = postprocess(output)             # โ† differs by domain

model(input) just takes a PyTorch tensor and returns a tensor. The challenge is converting raw data into that tensor.

Level 1: Text โ€” Tokenization (โ˜…โ˜†โ˜†โ˜†โ˜†)

Easiest. Start here.

tokens = tokenizer("Hello world")["input_ids"]
# โ†’ [15496, 995]  โ€” lookup table, that's it

Why easy: 1D sequence, discrete integers, deterministic, tiny data.

Level 2: Tabular Data โ€” Normalize (โ˜…โ˜…โ˜†โ˜†โ˜†)

features = (features - features.mean()) / features.std()

Already numbers. Just scale them.

Level 3: Image โ€” Resize + Normalize (โ˜…โ˜…โ˜†โ˜†โ˜†)

preprocess = transforms.Compose([Resize(224), CenterCrop(224), ToTensor(), Normalize(...)])
tensor = preprocess(Image.open("cat.jpg"))  # [3, 224, 224]

Why harder than text: continuous values, larger data, need normalization stats.

Level 4: Audio โ€” STFT + Mel (โ˜…โ˜…โ˜…โ˜…โ˜†)

mel_spec = MelSpectrogram(sr=44100, n_fft=2048, n_mels=80)(wav)
log_mel = torch.log(mel_spec + 1e-9)  # [80, time_frames]

Why hard: need Fourier transform understanding, model-specific hyperparameters, non-intuitive frequency domain.

Level 5: Video โ€” Frames + Temporal (โ˜…โ˜…โ˜…โ˜…โ˜†)

frames = video[sampled_indices]  # [16, 3, 224, 224]

Image preprocessing ร— N frames + temporal dimension + sampling strategy.

Level 6: 3D / Point Clouds (โ˜…โ˜…โ˜…โ˜…โ˜…)

points -= centroid; points /= max_dist  # normalize
points = farthest_point_sampling(points, 1024)  # [1024, 3]

Hardest: irregular data, order-invariant, coordinate alignment, immature ecosystem.

Study Order

  1. Text โ†’ 2. Image โ†’ 3. Audio โ†’ 4. Video โ†’ 5. 3D

Text has the shortest distance to model(input). That's why it's the best starting point.

Key Concepts

1

Text (โ˜…โ˜†โ˜†โ˜†โ˜†) โ€” tokenization is a lookup table. String โ†’ integer array. Easiest

2

Image (โ˜…โ˜…โ˜†โ˜†โ˜†) โ€” resize + normalize. Convert to [3, 224, 224] tensor

3

Audio (โ˜…โ˜…โ˜…โ˜…โ˜†) โ€” STFT + mel filter bank + log scale. Requires Fourier transform understanding

4

Video (โ˜…โ˜…โ˜…โ˜…โ˜†) โ€” frame sampling + image preprocessing ร— N frames + temporal dimension

5

3D (โ˜…โ˜…โ˜…โ˜…โ˜…) โ€” point cloud normalization + FPS + coordinate alignment. Hardest

Use Cases

AI study roadmap โ€” study in order: text โ†’ image โ†’ audio โ†’ video โ†’ 3D Project difficulty assessment โ€” predict preprocessing complexity from data type Preprocessing pipeline design โ€” understand standard preprocessing patterns per domain

References

Related

๐ŸŽฏ
input โ†’ pre-trained model โ†’ output โ€” The Essence of AI Inference in One Line
Forget STFT and masking โ€” the model does exactly one thing: take input, produce output
→
← ๐ŸŽฏ input โ†’ pre-trained model โ†’ output โ€” The Essence of AI Inference in One Line ๐Ÿ“ Text AI Pre-trained Model Catalog โ€” 12 Things You Can Do in 3 Lines of Code →