Meta put the Llama 2 weights on a public download server in July 2023, and something shifted in the economics of artificial intelligence. Within a year, a graduate student with a single second-hand RTX 3090 could run a model that would have cost tens of millions of dollars to train. A team in Nairobi could fine-tune it for Swahili without asking anyone’s permission. That is the practical promise of open source AI, and it is far messier and more interesting than the marketing decks suggest.
What “open source” actually means in AI
In ordinary software, open source has a clear meaning: you get the source code, a licence that lets you modify and redistribute it, and enough documentation to rebuild it yourself. AI breaks that template at exactly one point. The training data and the compute.
Almost everything sold as open source AI is really open weights. You get a file of a few billion floating-point numbers plus the code to run them. You do not get the dataset. You usually cannot reproduce the training run, and nobody is going to hand you 25,000 H100s for a weekend.
The Open Source AI Definition, finalised by the OSI in October 2024, tried to draw a line. It asks for enough detail about training data that a skilled person could build a substantially equivalent system. Very few releases clear that bar. Meta’s Llama licence has never been OSI-approved, partly because of its acceptable-use policy and the clause about companies with more than 700 million monthly users.
None of that makes open weights useless. It just means you should know which of the two things you are actually getting before you build a business on top of it.
The models worth knowing
The catalogue changes monthly, but a handful of families have become the default reference points.
- Llama 3.1 and 3.3 from Meta, in 8B, 70B and a 405B flagship. Distributed under a community licence with a commercial carve-out and a naming requirement.
- Mistral and Mixtral from the Paris-based lab, mostly Apache 2.0. A leaked 7B torrent in 2023 arguably started the whole trend, and Mistral has kept shipping small, sharp models since.
- Qwen 2.5 from Alibaba, spanning 0.5B to 72B, largely Apache 2.0. The small variants are unusually strong at code and arithmetic for their size.
- DeepSeek V3 and R1, released under MIT. R1 is the one that made people sit up, because a permissively licensed model was trading blows with closed frontier systems on maths benchmarks. It also forced a lot of conversations about what reasoning AI genuinely does better rather than what the demo reels claim.
- Gemma from Google, 2B to 27B, tuned for single-GPU and laptop use.
- Phi from Microsoft, small models that reason well above their weight class, though the claims about textbook-quality synthetic data deserve a sceptical read.
Size is not the only variable. A 14B model fine-tuned on your own support transcripts will beat a 70B generalist on your specific task almost every time, and it will cost a tenth as much to serve.
Why teams pick open weights
Cost is the headline, but it is rarely the whole story. Hosted APIs charge per token forever. A local model costs you a GPU once, then electricity. For a product handling a few hundred million tokens a month, the crossover point arrives faster than most finance teams expect.
The other reasons tend to be structural:
- Patient records, legal files and source code never leave your network.
- No rate limits on a launch day, and no vendor deprecating the model version your prompts were tuned against.
- Fine-tuning is a weekend job rather than a support ticket.
- Offline operation, which matters for field equipment, medical devices and anything on a ship.
The hardware reality check
This is where enthusiasm meets arithmetic. A model’s memory footprint is roughly its parameter count multiplied by the bytes per parameter, plus overhead for the context window.
At 4-bit quantisation, which is the usual sweet spot before quality visibly degrades, the numbers look like this: an 8B model needs about 5GB of VRAM, a 14B needs around 9GB, a 32B needs roughly 20GB, and a 70B wants 40GB or more. Add a couple of gigabytes for a long conversation. Consumer cards with 8GB of memory will comfortably run 8B models. Anything past 32B means a 24GB card, a pair of them, or a Mac with unified memory.
The tooling has become genuinely pleasant. Ollama gets you running in a single command. llama.cpp runs on almost anything, including a Raspberry Pi at painful speed. LM Studio gives you a point-and-click interface. vLLM is what you reach for when you need to serve dozens of concurrent users rather than one.
Where open models still fall short
Honesty helps here. The gap between the best open weights and the best closed systems has narrowed to a few months on benchmarks, but benchmarks are not the job.
Long-horizon tasks remain the weak spot. Give a model twelve sequential tool calls and a vague objective, and open models drop threads more often than frontier systems do. The failure modes are predictable: silently skipping a step, inventing a function argument, losing track of what it already tried. If your product depends on that kind of autonomy, read up on where agentic AI still breaks down before you promise it to a customer.
Multilingual nuance and safety behaviour are the other soft spots. Fine-tuning on English data tends to degrade performance in other languages, sometimes badly. Open models also arrive without the guardrails a hosted API applies on your behalf, which is freedom and liability in the same package.
Licences are where the surprises live
Read the licence before the benchmark table. Apache 2.0 and MIT mean what they say: use it, sell it, modify it, keep quiet about it. Llama’s community licence is more restrictive, with its acceptable-use terms and that monthly-active-user threshold that triggers a separate agreement. Gemma ships under Google’s own terms. Some Chinese releases carry usage restrictions that surprise Western legal teams.
There is also a subtler trap. A model’s weights may be permissive while its tokeniser or training data carry different obligations. If your legal review consists of checking the GitHub repo’s licence badge, you have not done legal review.
Getting a model running this afternoon
The distance between curiosity and a working local model is now about twenty minutes. Install Ollama, pull a model with one command, and you have an API on localhost that speaks the same shape as the big providers. Swap the base URL in your existing code and nothing else changes.
From there, the useful work begins. Pick a narrow task you already understand well, build a small evaluation set of twenty to fifty real examples with known good answers, and score each model you try against it. Prompt engineering on vibes is how people end up shipping a 70B model to do a job an 8B could handle. If you want a structured starting point, this walkthrough on using AI GPT for real work sets out a repeatable evaluation loop that transfers directly to open models.
Then build something small and real. An action-item extractor from meeting notes is a good first project: it needs no database, the output is easy to verify, and you will learn more about quantisation, context limits and prompt fragility in an afternoon than in a month of reading. There is a full tutorial on starting AI programming with a working extractor if you want the code laid out step by step.
Start with an 8B model, measure where it fails, and only then reach for something bigger. Most teams discover their bottleneck was the task definition, not the parameter count.