Local LLM Tool Catalog โ Running AI on Your Own Machine
From Ollama and llama.cpp to LM Studio and vLLM โ organized by use case
Cloud APIs are convenient. But environments where you can't send company docs to GPT, personal projects where monthly API costs add up, coding on a plane โ moments where local LLMs make sense come up more than you'd think.
Ollama โ Install This First
brew install ollama
ollama run llama3.1
That's it. Pull models like Docker and use them immediately. Auto-starts an OpenAI-compatible API at localhost:11434. Change base_url in existing OpenAI SDK code and you're running local.
M1+ Macs get Metal acceleration automatically. 8GB RAM handles 7B models, 16GB handles 13B comfortably.
llama.cpp โ The Engine Under Ollama
The inference engine Ollama uses internally. C/C++, runs GGUF quantized models on CPU. Supports Apple Silicon, CUDA, ROCm, Vulkan. You rarely need to touch it directly unless you need custom quantization or model conversion.
LM Studio โ For Those Who Need a GUI
Search HuggingFace models, one-click download, chat UI ready. Non-developers can use it. Uses llama.cpp internally. Can also run an OpenAI-compatible local server.
GPT4All โ Offline Desktop App
Desktop app by Nomic. Install and chat immediately. Can embed local documents (PDF, txt) for RAG search. More limited than Ollama, but better for non-developers who just want to chat.
vLLM โ Production Serving
For serving LLMs on your own GPU server. PagedAttention for memory efficiency, continuous batching for concurrent requests. NVIDIA GPU required. The answer when multiple people need to use the same model.
Text Generation WebUI โ All-in-One Experimentation
Gradio-based web UI. Model loading, chat, parameter tuning, LoRA, API serving โ everything. Multiple backends. Powerful but complex.
LocalAI โ OpenAI API Drop-in Replacement
Provides identical OpenAI API endpoints locally. Not just LLMs โ image generation (Stable Diffusion), speech recognition (Whisper), TTS, embeddings all local.
Which Tool Should You Use?
Starting out: Ollama. Five minutes.
Coding assistant: Ollama + Continue (VS Code) or LM Studio + Continue.
Non-developer/offline chat: LM Studio or GPT4All.
Team serving on GPU server: vLLM.
Switching existing OpenAI code to local: Ollama (simple) or LocalAI (images/audio too).
Most cases: start with Ollama, expand as needed.
Key Concepts
Ollama โ brew install โ ollama run to start local LLM in 5 minutes
llama.cpp โ engine inside Ollama. Runs GGUF quantized models on CPU/GPU
LM Studio / GPT4All โ GUI model management + chat. Non-developers welcome
vLLM โ production serving on GPU servers. PagedAttention for memory efficiency
LocalAI โ fully OpenAI API compatible. LLM + image + audio + embeddings all local
Pros
- ✓ Data stays local โ safe for internal docs and personal information
- ✓ Zero API cost โ unlimited inference for just electricity
- ✓ Works offline โ planes, subways, VPN-blocked environments
- ✓ Full customization โ quantization level, system prompt, LoRA application under full control
Cons
- ✗ Performance ceiling โ GPT-4o/Claude level is unrealistic locally
- ✗ RAM/VRAM constraints โ bigger models need more hardware
- ✗ Non-English performance โ many models still have a gap vs English