Model Merging โ The Only Way to Actually "Add" Two Models Together
Korean model + coding model = model that codes in Korean? Merge by averaging weights
The Only True "Adding" in AI
Fine-tuning, LoRA, RAG โ none are really "adding." Model merging is actually merging.
Model A weights: [0.50, -0.30, 0.80, ...]
Model B weights: [0.40, -0.10, 0.60, ...]
Merged: [0.45, -0.20, 0.70, ...] โ average
Average 66M numbers one by one. No training needed. Just arithmetic.
Why Does This Work?
Two models fine-tuned from the same base have similar weights. Averaging their adjustments can combine both capabilities.
Simplest Merge Code
merged = {}
for key in weights_a:
merged[key] = (weights_a[key] + weights_b[key]) / 2
torch.save(merged, "merged.safetensors") # Done. No training. No GPU.
Limitations โ Not Guaranteed
Experimental. Works with same-base models on complementary domains. Fails across different architectures or very different domains.
Comparison
| Fine-tuning | LoRA | RAG | Merging | |
|---|---|---|---|---|
| Weight change | All | Partial | None | Average two models |
| Training | Yes | Yes | No | No |
| GPU | Yes | Yes | No | No |
| Guaranteed | High | High | High | Low (experimental) |
Key Concepts
Prepare 2 models fine-tuned from same base (e.g., Korean model + coding model)
Load weights from both models
Average all 66M numbers one by one โ (A + B) / 2
Save merged weights โ no training, no GPU needed
Test โ check if both capabilities merged (not guaranteed)