Last month I downloaded a 4.7GB file, ran a one-line installer, and had a chatbot answering questions on my laptop with the Wi-Fi switched off. No account. No API key. No running bill. It produced about 28 tokens a second on a three-year-old machine, which is comfortably faster than I read.
That’s a local AI chatbot: a language model running on your own hardware instead of someone else’s data centre. Same underlying technology as ChatGPT or Claude, packaged very differently, with a very different set of trade-offs. Most of those trade-offs are excellent. A few will drive you up the wall.
What “local” actually means in practice
Three pieces sit between you and a reply.
The runtime loads the model and does the maths. Ollama and LM Studio are the two most common on desktop; llama.cpp is the engine underneath both; GPT4All and Jan are friendlier wrappers around the same idea.
The weights are the model itself, usually a single file in GGUF format. Llama 3.1 8B at 4-bit quantisation lands around 4.7GB. A 70B model in the same format needs roughly 40GB.
The front-end is whatever chat window you type into. It might be the runtime’s built-in one, or something like Open WebUI if you want it to feel like a proper app.
Everything happens on your disk and your silicon. Nothing leaves the machine unless you deliberately configure it to.
The hardware reality check
Memory is the whole game. A 4-bit quantised model needs roughly its file size in free RAM or VRAM, plus a couple of gigabytes of overhead for the context window.
- 8GB RAM, no GPU: 3B models, and slowly. Fine for summarising and rewriting, painful for anything longer.
- 16GB RAM or a 16GB Apple Silicon Mac: 7B to 9B models at 15-30 tokens/sec. The practical sweet spot for most people.
- 12GB VRAM (RTX 3060 and similar): 8B models fully on the GPU, 14B models with partial offload.
- 24GB+ VRAM or 32GB+ unified memory: 30B-class models, noticeably better reasoning, still nowhere near frontier quality.
Apple Silicon punches above its weight here because the GPU and CPU draw from the same memory pool. A MacBook with 18GB of unified memory will run an 8B model faster than plenty of Windows laptops with dedicated graphics cards.
Privacy is the real selling point, with some asterisks
If confidentiality is why you’re reading this, the case is genuinely strong. Client contracts, medical notes, unreleased code, a diary you’d rather nobody indexed. None of it crosses a network boundary, so none of it can land in a training set, a support ticket, or a breach notification.
The asterisks matter, though. Your prompts and the model’s replies get written to disk in chat history by default in almost every tool. If the laptop walks, that history goes with it, encryption or not. Local also stops being anonymous the moment you paste the output into a cloud service. There’s a useful breakdown of where the boundaries really sit in what actually stays private with AI chatbots, and it’s worth reading before you treat “runs locally” as a blanket guarantee.
Where local models get frustrating
The knowledge cutoff is the big one. An 8B model trained up to a certain date simply doesn’t know what happened afterwards, and unlike a cloud assistant it can’t look anything up unless you build that in yourself. Ask about a release from last month and it will either admit ignorance or invent something plausible. The second option is worse.
You can wire up web access through tooling, and the behaviour shifts in ways that aren’t always obvious. If you’re considering it, giving a chatbot internet access covers what improves and what quietly degrades.
Small models also follow complicated instructions less reliably. Hand a 7B model a five-part formatting request and expect to repeat yourself twice. Hand it a two-part request and it’s often indistinguishable from something far larger.
Which models are worth downloading
Llama 3.1 8B is the dependable default: good writing, solid instruction following, wide support. Qwen 2.5 7B and 14B are strong at code and structured output. Gemma 2 9B has a pleasant tone for editing prose. Phi-3.5 Mini is tiny and quick for classification and extraction work.
Then there are the reasoning models, which think before they answer. The distilled DeepSeek variants run surprisingly well at 7B and 8B, and they’re better at multi-step maths and logic than their size suggests. DeepSeek’s open-weight models are a good place to start if you want to see what open reasoning looks like without a GPU farm.
Setup takes about twenty minutes
Install Ollama, open a terminal, run ollama run llama3.1:8b. It pulls the weights and drops you into a chat prompt. That’s the entire process on macOS, Windows and Linux.
LM Studio is the better route if you’d rather not touch a terminal. Search a model inside the app, click download, load it, chat. It also exposes a local server on port 1234 that speaks the OpenAI API format, which means most existing code and tools will talk to it after changing one base URL.
Expect the first load to take 10-30 seconds while weights move into memory. Replies after that start almost instantly.
The economics, and where the savings actually land
Cloud subscriptions are predictable; local is a lump sum. A $20/month plan costs $240 a year. A machine that runs an 8B model comfortably costs rather more than that, so buying hardware specifically to replace a subscription rarely pays off on price alone.
The maths changes when you already own the machine, or when your usage is heavy. Someone pushing 200 documents a week through an API can easily spend more on tokens than the electricity and depreciation of a laptop they already have. Running a local model flat out adds maybe 40-60W to a desktop’s draw, which is pennies an hour.
It changes again when the work is confidential enough that the cloud option was never really on the table.
What people actually use them for
In practice the jobs are modest and repetitive. Rewriting an email so it doesn’t sound passive-aggressive. Turning meeting notes into a task list. Explaining a stack trace. Summarising a 40-page PDF you’d rather not upload anywhere.
Developers use them to make sense of unfamiliar code in a private repo. Writers use them for structural feedback on drafts. A few people I know run one purely as an offline journal companion on a laptop that never joins a network.
For heavier, messier work such as campaign planning or multi-source research, cloud tools still win on quality by a wide margin. If you’re weighing up where the time savings really are, there’s a grounded look at AI tools for marketing that makes the split fairly obvious.
Be choosy about what you install, too. Most local AI apps launching each month are thin wrappers around the same two or three runtimes, and a month of real use usually separates the keepers from the clutter, as this long test of new AI tools found.
The hybrid setup that actually sticks
Most people don’t end up choosing. They keep a cloud assistant for hard reasoning, current information and long documents, and a local model for anything sensitive, anything they’d rather not have logged on someone else’s server, and anything they want to run on a plane.
The glue is small. Point your editor, notes app or a script at the local server endpoint and route requests by sensitivity rather than habit. Confidential material never leaves the house; everything else gets the strongest model you can afford. Once that’s in place, “local or cloud” stops being an identity and becomes a routing decision you make in about two seconds.