Most of the AI you talk to runs in someone else’s data centre. It doesn’t have to. Open models have caught up enough that you can run a genuinely useful AI on your own machine — and tools like Ollama make it a single command. This page is the plain-language version: why you would, what you need, and how to get from nothing to chatting on your own hardware.
Why run your own?
Your data stays yours
Everything stays on your machine. No prompts leaving, no logs collecting on someone else’s servers. This is why local models are the standard answer for sensitive work.
No bill per token
The hardware is the one-time cost. After that it’s free — unlimited chat, unlimited API calls, no metering.
Works offline
No internet needed. Useful on planes, boats, and sites where data simply cannot leave the building.
Total control
No rate limits, no silent version rollouts, nothing in the terms of service. When the model changes, that’s your call.
Two honest caveats: the very top frontier models are still bigger than a single graphics card can hold, and a laptop alone won’t comfortably run the middle tier. A local model is a real choice — not yet a free upgrade.
The two tools you need
Ollama
Ollama runs open models on macOS, Windows and Linux. It handles the download, the memory setup and the chat interface — and exposes a simple API that any tool can use. Install from ollama.com/download, then:
- Run
ollamain a terminal — it opens an interactive menu. - Start a chat with
ollama run gemma4. - Need a bigger model without the download?
ollama run gemma4:cloudruns it in Ollama’s cloud.
ollama run gemma4
Every model and its command is listed at ollama.com/library.
Unsloth
Unsloth is a free, open-source app that runs and trains AI models on your own hardware. The fastest route: install it, open the browser UI, download a model — and either start chatting or start training. Or download the Unsloth Desktop app from unsloth.ai/download (macOS, Windows, Linux).
curl -fsSL https://unsloth.ai/install.sh | sh
Then open http://127.0.0.1:8888 in your browser. Unsloth’s own claim: it trains 500+ models roughly 2x faster with about 70% less VRAM, with no accuracy loss.
What you need
The one number that matters is GPU memory (VRAM). Models are measured in billions of parameters; a quantized model needs roughly 1 GB of VRAM per billion parameters at 8 bits. This table is the practical summary:
| VRAM | Model sizes | Example |
|---|---|---|
| 4–6 GB | 3–4B | Gemma 3 4B, Phi-4 Mini |
| 8–12 GB | 7–8B | Qwen 3 8B, Llama 3.1 8B — the sweet spot for most people |
| 16–24 GB | 13–32B | Qwen 3 32B, Gemma 3 27B |
| 48 GB+ | 70B+ | Llama 3.3 70B — workstation class |
No GPU at all? Small models still run on the CPU — slowly, but they run. If a model doesn’t fit in VRAM and spills into system RAM, speed collapses, so that memory check above is the simplest filter there is.
Teach it your way
When a stock model isn’t quite right — your tone, your docs, your tool conventions — you can teach it. Fine-tuning takes a ready-made model and trains it further on your own examples. It needs far less data and compute than training a model from scratch, especially with LoRA adapters, which are a few tens of megabytes and are famous for it. Unsloth’s memory trick (training with much less VRAM) means a free Google Colab T4 (15 GB GPU) is enough for small and medium models — no GPU on your desk required.
- 1. Pick a base model. Qwen 3.5, Gemma 4, gpt-oss and the rest — all in Unsloth’s model hub.
- 2. Add LoRA adapters. A tiny, trainable layer that records what changed — no need to retouch the whole model.
- 3. Train on your examples. Watch loss and GPU usage live in Unsloth’s UI.
- 4. Export to GGUF and run it. Unsloth’s Dynamic 3.0 GGUFs drop straight into Ollama or llama.cpp.
Point your agent at it
A local model is also a safe basis for an agent: no data leaves the machine. With Unsloth running, one command points your existing agent at your own model:
unsloth start claude
unsloth start codex
unsloth start hermes
unsloth start opencode
Unsloth also exposes an OpenAI-compatible API, so any app that speaks that format can point at your local model instead of the cloud. Build the rest of your agent setup with the guide to AI Harnesses.
Pick a model
Open models are anyone’s to run and everyone’s to compare. A few current names from the Ollama library:
| Model family | Who makes it | Why it stands out |
|---|---|---|
| Gemma 4 | Frontier-level for its size; reasoning, agentic work, coding | |
| Qwen 3.5 | Alibaba | Multimodal family, 100+ languages |
| Qwen 3 Coder | Alibaba | Long context, agentic coding |
| gpt-oss | OpenAI | Open-weight reasoning and agentic models |
| Muse Glimmer | Meta | 30B, Apache 2.0, runs on a single GPU — built for always-on local agents |
| Phi 4 | Microsoft | Compact 14B, strong for its size |
You don’t have to choose on reputation: each page at ollama.com/library shows sizes, tags and the exact command, and trying them is free.