Whisper chewed through a 47-minute interview on my laptop in just under four minutes last week. No account, no upload, no per-minute billing. That’s the quiet appeal of open source AI tools: the work happens on hardware you already own, and nobody can change the terms on you between Tuesday and Thursday.
The category also has a hype problem in both directions. Some people treat open weights as a moral cause. Others dismiss them as a hobbyist’s toy that can’t touch the frontier models. Neither framing helps if you just want to get something done. So here’s a grounded look at what’s worth installing, what it costs, and where the licensing gets messy.
Why the gap closed so fast
Two years ago, running a capable language model locally meant compromises that weren’t worth making. A 7B model could write passable marketing copy and then invent a citation two sentences later.
Then Meta published Llama weights, Mistral shipped models under Apache 2.0, and labs in China, particularly Alibaba’s Qwen team and DeepSeek, started releasing weights that traded blows with closed models on public benchmarks. DeepSeek-V3 was trained for a reported $5.6 million and scored competitively against systems that reportedly cost ten figures to build.
The practical effect: the ceiling of what you can run on a single consumer GPU moved from embarrassing to genuinely useful in about eighteen months. The floor, meaning what runs on a laptop with no discrete GPU at all, moved from toy to fine for summarising and drafting.
The models worth your time
Text and reasoning
Llama 3.1 and 3.3 are the safest defaults because tooling support is everywhere and every tutorial you’ll find assumes them. Qwen 2.5 is the better pick if you want range: the family spans 0.5B to 72B parameters, several sizes ship under Apache 2.0, and the multilingual performance is strong. Mistral’s Small and NeMo models are efficient at 12B and 22B respectively. For reasoning-heavy work, the DeepSeek-R1 distills cram chain-of-thought behaviour into 7B and 14B packages that fit on modest hardware.
Images
The Stable Diffusion lineage remains the backbone here, though Flux.1 [schnell] under Apache 2.0 has become the default for anyone who wants speed. Note the licence carefully: some Flux variants and older SD checkpoints are non-commercial. Image generation is also the most hardware-hungry thing in this entire article. A 12GB card is the realistic entry point.
Audio
Whisper is the undisputed champion. MIT licensed, runs on CPU, handles roughly 99 languages, and the large-v3 checkpoint is accurate enough that I’ve stopped double-checking meeting transcripts. For speech synthesis, Piper is fast and light, and Coqui’s XTTS is archived but still widely used for voice cloning.
Tooling: the part that makes them usable
Raw weights are a folder of files. You need a runtime, and the runtime decides whether the experience feels like software or like a science project.
- Ollama handles model downloads and serves an OpenAI-compatible API on port 11434. One command and you’re running.
- llama.cpp is the engine underneath half the others, and the reason GGUF quantisation became the standard format.
- LM Studio gives you the same capability with a graphical interface, which matters if you’re handing a machine to a non-technical colleague.
- vLLM is what you reach for when several people need to hit the same model and throughput becomes the problem.
- ComfyUI turns image generation into a node graph. Ugly at first, wildly powerful once it clicks.
- Hugging Face is where every one of these models lives, along with Spaces demos you can try before downloading 40GB.
If your day job involves code, the AI tools that earn their place in a developer workflow are worth a look, because local models change the calculus on autocomplete and refactoring more than most people expect.
What “free” actually costs
Nothing about this is free, it’s just paid for differently. The numbers that matter:
An 8B model at 4-bit quantisation needs around 5GB of VRAM, so an RTX 3060 with 12GB runs it comfortably. A 32B model at the same quantisation wants roughly 20GB, which means a 24GB card like the RTX 3090 or 4090, or a Mac with 32GB or more of unified memory. A 70B model needs about 40GB, and that’s where consumer hardware stops being practical.
Electricity is a rounding error by comparison. A 350W card running four hours a day comes to roughly 1.4 kWh, so about 20 to 30 pence depending on your tariff. The real cost is your time, and the upgrade you’ll inevitably want.
Before you spend £1,500 on a GPU, it’s worth asking whether the subscription you’d be replacing is earning its keep in the first place. This breakdown of which AI productivity tools justify a subscription makes that case bluntly, and the arithmetic usually favours renting until you hit real volume.
Which is why the hosted middle ground exists. Together, Fireworks, Groq and OpenRouter all serve open weights at a fraction of closed-model API prices, with no hardware to buy.
Licensing: open weights are not open source
This distinction trips people up constantly. The Open Source AI Definition, finalised by the OSI in October 2024, requires data information sufficient for someone to rebuild a comparable system. Plenty of popular models don’t clear that bar, so they’re open weights rather than open source in any strict sense.
Llama’s community licence is the best-known example. You can use it commercially, but there’s a clause revoking the licence for products with more than 700 million monthly active users, plus naming and attribution requirements. Apache 2.0 and MIT models, including much of the Qwen and Mistral line, carry no such strings. Read the licence before you build a product on top of a checkpoint, especially for image models where non-commercial terms hide in the fine print.
Where open models genuinely win
Privacy is the obvious one, and it isn’t theoretical. Legal teams, clinics and HR departments can’t paste contracts or patient notes into a third-party API. A local model sidesteps the whole conversation, and for a lot of organisations that’s the entire business case.
Fine-tuning is the second. You can take an 8B model, train it on five years of your own support tickets, and get something that outperforms a frontier model on your narrow task at a fraction of the inference cost. Try that with a closed API and you’re stuck with prompt engineering.
Cost at scale is the third, though the crossover point arrives later than enthusiasts claim. Self-hosting starts to make sense once your API bill crosses a few hundred pounds a month and stays there. The economics of when to build versus buy are covered well in this piece on AI tools for entrepreneurs building a stack.
Then there’s the boring advantage nobody mentions: stability. A pinned model version doesn’t get deprecated, rate-limited, or quietly retuned by a vendor on a Friday afternoon.
Where they still fall short
Frontier reasoning is the honest gap. On genuinely hard multi-step problems, the top closed models still pull ahead, and no amount of local enthusiasm changes that. Long-context reliability degrades faster too: a local model may claim a 128k window, but quality often sags well before that.
Agentic tool use and computer control remain rough. So does support. When something breaks at 11pm, you have a GitHub issue thread and a Discord server, not a status page and an SLA.
Start with one job, not a platform
The most common mistake is treating this as an infrastructure project. You don’t need a homelab. You need one repeated task and twenty minutes.
Pick the task first. If it’s transcription, install Whisper and stop there. If it’s drafting, install Ollama, run ollama pull qwen2.5:7b, and see whether the output clears your quality bar within a week. Measure how often you actually reach for it. If the answer is twice, you’ve learned something cheap.
On weaker hardware, be realistic about what’s possible. A laptop with no discrete GPU will run a 3B or 4B model at tolerable speed and nothing larger. This short list of AI tools for students on ordinary laptops assumes exactly those constraints and doesn’t pretend otherwise.
And if you’re still deciding whether a given task is even worth automating, try it first with something disposable. A handful of AI tools you can use without signing up will tell you in ten minutes whether the workflow is real or imaginary, before you commit a single gigabyte of disk space to a model you’ll never open again.