In February 2024, the Allen Institute for AI uploaded something unusual to Hugging Face. Not just a model, but the training code, the logs, the intermediate checkpoints, and the 3-trillion-token Dolma dataset that produced it. Anyone with a decent broadband connection could pull down the entire recipe and take it apart on their own hardware.
That release did more to demystify model training than a decade of papers. It also exposed how loosely the phrase “open source” gets used in AI. Plenty of models are downloadable. Very few are actually reproducible.
What open source AI training actually covers
Three separate things get bundled together, and conflating them causes most of the confusion.
- Weights: the finished parameters. Runnable and fine-tunable, but frozen. You can build on them; you can’t see how they were made.
- Training code: the scripts, configs, optimiser settings and parallelism tricks that turn raw text into a working model.
- Data: the corpus itself, plus the filtering rules, the deduplication, and the order batches were fed in.
Most so-called open releases give you the first and stop. Meta’s Llama models ship weights and an architecture paper, but the 15 trillion tokens behind Llama 3 stay private. Mistral’s original 7B landed under Apache 2.0, which is genuinely permissive for weights, while the data recipe stayed in the vault.
Full transparency is rarer, and considerably more interesting.
The projects that hand you the whole recipe
OLMo and Dolma
AI2’s OLMo 2, released in late 2024 in 7B and 13B sizes, is the cleanest example available. You get the weights, the full training stack built on PyTorch and FSDP, the Dolma corpus, checkpoints saved throughout the run, and a paper that documents what went wrong along the way. If you want to reproduce a pretraining run end to end, start here.
Pythia
EleutherAI trained models from 70M to 12B parameters on the Pile and kept 154 checkpoints for every one of them. That means you can watch a model acquire grammar, then arithmetic, then factual recall, step by step. For anyone studying how capabilities emerge, it’s the closest thing to a lab notebook that exists.
The data layer: RedPajama, FineWeb, Dolma
Together’s RedPajama reproduced the LLaMA training mix at 1.2 trillion tokens. Hugging Face’s FineWeb pushed past 15 trillion tokens of filtered Common Crawl, with the filters documented. Data is the part of the pipeline that quietly decides how good a model gets, and these collections are free to download.
nanoGPT and llm.c for the mechanics
Andrej Karpathy’s nanoGPT is a few hundred lines of PyTorch that trains a character-level model on Shakespeare using a laptop. His llm.c project reproduces GPT-2 124M in raw C and CUDA for roughly $20 of rented compute. Neither produces a useful assistant. Both make the rest of the literature readable in an afternoon.
Pretraining or fine-tuning: the call most people get wrong
Almost everyone who says they want to train a model actually wants to fine-tune one. The difference in cost is roughly three orders of magnitude.
- Pretraining builds a model from random weights. Trillions of tokens, thousands of GPUs, weeks of wall-clock time.
- Fine-tuning adapts an existing model to your data. A few thousand examples, one GPU, a few hours.
LoRA and QLoRA shrank the second category dramatically. The QLoRA paper fine-tuned a 65B parameter model on a single 48GB GPU by quantising the base weights to 4-bit. Today an 8B model trains comfortably on a 24GB RTX 4090 using Unsloth or Axolotl, and renting a single A100 on Lambda or RunPod for a weekend costs less than most people’s coffee budget for a month. For a support triage classifier, a domain-specific writing assistant, or a model that has to read your internal tickets, that really is the whole job.
The hardware math nobody puts on the landing page
Pretraining is a different sport with a different price list. Meta reported that Llama 3 405B consumed 30.84 million H100 hours. At two dollars an hour that’s over $60 million in compute alone, before you pay anyone who knows how to stop sixteen thousand accelerators from falling over mid-run.
The arithmetic is at least simple. Training compute in FLOPs works out to roughly 6 × parameters × tokens. A 7B model on a trillion tokens needs about 4.2 × 10^19 FLOPs, which in practice lands somewhere in the tens of thousands of GPU hours once you account for real-world efficiency. That’s a five-figure cloud bill at spot prices, and a six-figure one if a node dies at week three and you restart from an old checkpoint.
The licensing fine print
Open weights are not open source, whatever the marketing says. The Open Source Initiative published version 1.0 of its Open Source AI Definition in October 2024. It asks for enough data information that a skilled person could rebuild a substantially equivalent system. That bar rules out most models casually described as open.
Llama’s community licence adds a clause requiring a separate agreement with Meta once a product passes 700 million monthly users. Gemma arrives with a use policy that forbids certain applications outright. Apache 2.0 and MIT, which cover Mistral’s early 7B and much of the Qwen line, ask for little beyond attribution. Read the licence before you build a company on top of weights, not eighteen months later.
Learning this without burning cash
You don’t need a GPU to start. You need attention, tokenisation, the training loop, and a working sense of why learning rate schedules matter. Free material is everywhere; the hard part is picking a course you’ll actually finish, which is the premise behind this guide to the best free AI courses for students and the wider hunt for free AI classes worth your evenings.
If a vendor’s fingerprints help you trust the material, the free Microsoft AI courses cover the PyTorch and Azure side of the stack in reasonable depth. And if you learn by building rather than watching, the free AI coding courses that walk you through implementing a transformer will teach you more in a weekend than a month of lectures. Write the training loop yourself once, badly, and every config file afterwards stops being magic.
Where distributed training is heading
The most interesting shift is that pretraining is being pulled apart and spread out. Prime Intellect trained a 10B parameter model across three continents in late 2024, coordinating rented and donated GPUs over a decentralised protocol. Nous Research’s DisTrO work attacks the bandwidth bottleneck that makes this hard, compressing the gradient exchange between nodes by orders of magnitude.
If that line of work holds up, the five-figure pretraining bill becomes something a university department or a determined group of strangers can split. Until then, the honest path for most people runs through fine-tuning an open model, reading somebody else’s training code until it stops looking like sorcery, and getting comfortable with data pipelines long before touching a cluster. OLMo’s repository is public. Dolma downloads in an afternoon. The recipe is sitting there, unopened.