A chat window opens, you type a question, and an answer streams back. Nothing remarkable about that. What is remarkable is that the model generating those words never touches the internet. It runs on your laptop, using your RAM, and keeps working when the hotel Wi-Fi doesn’t.
That’s the idea behind Jan AI, a free, open-source desktop assistant built by Menlo Research. It looks like ChatGPT, behaves like ChatGPT, and is powered entirely by models sitting on your own hard drive. For a growing number of people, that combination solves problems the cloud version can’t.
What Jan AI actually is
Jan is an application you install rather than a service you subscribe to. There are builds for Windows, macOS and Linux. You pick a model from the built-in hub, download it, and start chatting. No account, no rate limits, no monthly bill. The source code is public under an Apache 2.0 licence, so anyone can audit it, fork it, or ship their own version.
Inference runs through Cortex, Jan’s own engine, which loads GGUF files — the compressed format popularised by llama.cpp. That’s how a 4GB download can turn a mid-range laptop into a workable AI assistant.
The model hub is the part newcomers underestimate. Llama, Qwen, Mistral, Gemma, DeepSeek, plus a long tail of community fine-tunes, all install in a couple of clicks. New ones land every few weeks, which makes keeping up feel like its own hobby, and a decent illustration of how fast the AI frontier keeps moving.
Why local changes the deal
Three things shift when the weights live on your SSD instead of in a data centre.
Privacy becomes a fact rather than a policy. Prompts never leave the machine. For lawyers, clinicians and developers working with proprietary code, that removes an entire category of approval headache.
Costs stop scaling with use. A heavy week of experimentation costs nothing. Anyone who has burned through a cloud API budget in three days knows how fast token bills add up.
It works offline. Flights, trains, rural jobsites, air-gapped networks. A local model doesn’t care where you are.
There’s a fourth, quieter benefit. Cloud providers deprecate models, revise terms and reshuffle teams, and the safety reckoning inside OpenAI showed how quickly priorities at a frontier lab can change. A model file on your drive doesn’t get sunset out from under your workflow.
Getting it running without wasting an afternoon
Match the model to your machine
RAM is the constraint that matters most. Rough guide:
- 8GB: 3B–4B parameter models at Q4 quantisation. Fine for summarising, drafting and simple chat.
- 16GB: 7B–8B models at Q4_K_M, the sweet spot most people should start with.
- 32GB or more: 13B to 30B models, noticeably better at reasoning and code.
- Apple Silicon: Metal acceleration works well, and unified memory means an M-series chip with 16GB handles what a cheaper PC needs a dedicated GPU for.
- Discrete GPU with 8GB+ VRAM: the fastest option. Let Jan offload as many layers as the card allows.
Bigger isn’t automatically better. A 7B model that answers in two seconds beats a 70B model that swaps to disk and takes four minutes. Watch the context length too, because a 32K context window eats far more memory than the default.
How fast does it actually feel?
On an M2 MacBook Air with 16GB, a Llama 3.1 8B model at Q4 usually generates somewhere between 15 and 25 tokens per second. That’s faster than most people read, so replies feel immediate. The same machine trying to run a 70B model will crawl, or stall completely.
Two numbers matter: time to first token, and generation speed. The first depends on how quickly the model loads and digests your prompt. The second depends on hardware and model size. If chat feels sluggish, drop to a smaller quantisation before blaming the software.
The local API is the sleeper feature
Jan runs an OpenAI-compatible server, usually on 127.0.0.1:1337. That single detail turns it from a chat app into infrastructure. You can point VS Code extensions, coding assistants, LangChain scripts or your own half-finished tools at that endpoint, and they’ll talk to it exactly as they would to a paid API.
Recent versions also support MCP, the protocol that lets models call external tools and data sources. It’s the same agentic direction the whole industry is heading, which raises an obvious question for anyone building on top of it: will the agent repeat a successful run, or was that one good result a fluke? Local models tend to be less consistent at multi-step tool use than their frontier counterparts, so test anything important more than once.
Where local models still lose
Being honest about the tradeoffs matters, because the gap hasn’t closed.
Frontier cloud models handle long documents, messy reasoning chains and complicated refactors better than anything you can squeeze into 16GB of RAM. They’re also improving fast, and OpenAI’s push for speed at the frontier suggests response time is becoming a competitive battleground in its own right.
What local models give you in exchange is control. No outage takes your assistant down, no provider decides your use case violates a policy, and nobody raises prices on you mid-project. The sensible setup for most people includes both: a local model for everyday drafting, private documents and offline work, plus a cloud subscription for the genuinely hard problems.
Five things worth trying in your first week
- Drop a 40-page PDF into a chat and ask for a one-page brief.
- Write a system prompt that turns your assistant into a picky editor for your own writing.
- Point a coding extension at the local server and see how it handles a real task.
- Switch off Wi-Fi and keep working, just to confirm everything is genuinely local.
- Run something silly. Hand a 3B model an absurd brief — the same energy as training a fly’s brain to pitch story ideas — and see what comes back. It costs nothing, which is the entire point.
Start with an 8B model at Q4_K_M and keep the app updated. The first message after launch takes a few seconds while the weights load into memory; after that it settles into something that feels like any other chat app. The difference is that the bill at the end of the month is zero, and nothing you typed went anywhere.