A partner emails over a 40-page merger agreement at 9pm and wants notes by morning. Pasting it into a chatbot would take thirty seconds. You hesitate, because you genuinely don’t know whether that document ends up in a training run, a support log, or a subprocessor’s storage bucket in another country.
That hesitation is the entire reason people go looking for a private AI chatbot. The trouble is that “private” gets used to describe at least four different setups, and only some of them match what you actually need.
The Four Promises Hidden Inside “Private”
When a vendor calls their chatbot private, they usually mean one of these things, and occasionally they’re happy for you to assume all four:
- Your prompts aren’t used for training. The company still receives and stores them, but promises not to feed them back into the model.
- Your prompts aren’t retained at all. Stricter. Logs are discarded once the request completes, usually with a short exception window for abuse monitoring.
- Your prompts never leave the device. The model runs on your laptop or phone. Nothing crosses the network.
- No third party is in the chain. You host the model on your own hardware or cloud account, so the only outside party is your infrastructure provider.
Only the last two are privacy in the architectural sense. The first two are contractual promises. Those are worth real money in a business context, but they depend entirely on the vendor keeping them, and on you being able to audit that.
Three Routes to a Private AI Chatbot
Run an open-weight model on your own machine
Tools like Ollama, LM Studio and llama.cpp turn this into a fifteen-minute project. You pull down a model such as Llama 3.1 8B or Qwen 2.5 7B and it runs locally: no account, no API key, no traffic once the download finishes. The chat interface is plainer than ChatGPT’s and the reasoning is noticeably weaker on hard problems. For summarising, drafting, extracting fields from a document, and rewriting, an 8B model is genuinely usable.
Pay for zero-retention terms on a hosted model
OpenAI, Anthropic, Google and the big cloud platforms all sell business tiers where API traffic isn’t used for training and retention is capped at roughly 30 days for abuse monitoring, with zero-retention options on some plans. You get the strongest models available and a contract that says the right things. What you don’t get is an architectural guarantee, and you pay per token.
Use an on-device assistant
Apple Intelligence, Gemini Nano on recent Pixel phones, and a growing crop of keyboard and notes apps ship small models in the 1B to 3B range. They handle sentence rewriting, quick summaries and notification triage entirely offline. Ask one to reason through a contract clause and you’ll hit the ceiling in about two prompts.
The Hardware Bill Nobody Mentions
Local models are free to download and expensive to run well. Rough numbers for what fits where:
- 8B model at 4-bit quantisation: about 5GB of weights, comfortable on 8GB of VRAM or a 16GB MacBook Air.
- 32B at 4-bit: roughly 20GB. You need a 24GB GPU or a 32GB unified-memory Mac.
- 70B at 4-bit: around 40GB. Two 24GB cards, or a Mac Studio with 64GB or more.
That last tier is where local output starts approaching hosted quality on complex work. A single top-end consumer GPU costs more than a couple of years of mid-tier API usage for most individuals, so the maths rarely favours buying hardware for privacy alone. If you’re weighing it up, why per-token pricing tends to cost more than the sticker suggests is a useful reality check before you spend.
Read the Retention Clause, Not the Badge
A “private mode” toggle is not a data processing agreement. Four things matter in the actual documentation: whether inputs are used for training by default or only on opt-in, how long logs persist, which subprocessors touch the data, and whether human reviewers can read flagged conversations.
Free tiers are the weakest link, because the trade for not paying is usually a broader licence over whatever you type. There’s a solid breakdown of that bargain in what free AI chatbots quietly take in exchange. Two questions cut through most marketing: “If I delete my account, when are my prompts actually gone?” and “Can a human at your company read this conversation?” Vague answers are answers.
What a Local Model Can’t Tell You
Run a model on your own machine and it knows nothing past its training cutoff. It can’t check a live price, verify a citation, or read a page you link to. That sounds minor until you ask for a current statistic and get a confident, fluent, completely invented number. Hosted models with browsing have their own trust problems, but the gap is real: what an AI chatbot with internet access actually changes is mostly about freshness trading against reliability.
Privacy Without Guardrails Is Just Isolation
Keeping data on your own box solves the leakage problem and creates a new one: nobody is watching what the model does with it. Connect a local model to your file system, your inbox or a database and you need limits on what it can read, what it can write, and which actions need a human yes first. The patterns that hold up are familiar ones, and designing architectural guardrails around AI agents covers them properly: scoped credentials, read-only defaults, approval steps before anything destructive.
A Data Triage Beats a Vendor Decision
Most teams don’t need one chatbot. They need a rule about which data goes where, decided once and written down. Three buckets cover almost every case:
- Public and low-stakes: marketing copy, brainstorming, open-source code snippets. Any hosted model, cheapest tier, no ceremony.
- Confidential but not regulated: internal strategy, unpublished pricing, customer feedback. Business tier with no-training terms and a retention cap, or a local 8B model if the task is simple.
- Regulated or contractually restricted: health records, legal documents under NDA, anything with personal data. Local model on a managed device, or don’t put it in a chatbot at all.
Write the buckets down, name who owns the decision, and revisit it when a new tool lands in your Slack workspace. The specific product matters far less than the habit of asking where the text goes before you press enter. A private AI chatbot is one piece of that picture, not a replacement for having a policy.
Before you commit to anything, test. Install Ollama on a work laptop, pull an 8B model, and spend twenty minutes on the three tasks you’d actually use it for. You’ll learn more about whether local inference is good enough for your work than any comparison table will tell you. If you’d rather experiment without creating accounts first, what you can get done in ten minutes without signing up is a sensible starting point.