The number looked right. It was specific, it had a decimal point, and ChatGPT Plus attributed it to a report I had half-heard of. I put it in a draft. Two days later a reader sent me the actual report, where the figure was 6.1%, not 16.1%. The model had not lied, exactly. It produced something that looked like an answer, and I let the confidence do the work of verification.
That is the whole problem with reviewing accuracy in an AI assistant. You cannot test it once and be done. You test it across dozens of ordinary tasks, note where it holds, then build habits around the places it doesn’t. What follows is what I found after several months of using Plus as a working tool rather than a demo.
What “accuracy” is actually made of
People use the word to mean three different things, and the three behave nothing alike.
- Recall. Does it know a date, a definition, the boiling point of a solvent?
- Reasoning. Given correct inputs, does it reach a correct conclusion?
- Calibration. Does it know, and admit, when it is out of its depth?
Recall gets all the attention because it is easy to spot when it fails. Reasoning is where the subscription earns its money. Calibration is the one almost nobody tests, and the one that costs people real money.
Where the model holds up under pressure
Hand it a mess and it does surprisingly well. I pasted 40 rows of invoicing data with two duplicate entries and asked it to find the mismatches. It caught both duplicates and a date-format error I had not noticed. Twelve months ago the same task produced a confident list of errors that did not exist.
Structured reasoning has improved the most. Ask why a delivery schedule keeps slipping and it will separate the constraint you can fix from the one you can’t. Ask it to debug a Python function and it usually finds the real fault rather than the one you described.
Summarising text you supply is close to reliable. Across 18 documents, from meeting transcripts to a 60-page vendor contract, nothing it wrote contradicted the source. That is a low bar, but plenty of tools miss it.
Web access helps with staleness, up to a point. The features I actually use every day are the boring ones: file upload, web search, and the slower reasoning mode. The flashier additions mostly change how an answer is presented, not whether it is right.
Where it still slips
The failure modes are consistent enough to name.
- Invented citations. Titles, authors, DOIs and page numbers can all be plausible and fictional in the same sentence.
- Arithmetic that survives the first look. Unit conversions and percentage changes are the weakest spot in my testing.
- Stale facts stated in the present tense. An old pricing tier, a discontinued feature, a person who has since changed jobs.
- Agreeing with your premise. If you ask why a policy failed, it rarely asks whether the policy failed.
- Silent drops in long documents. Middle sections get skimmed in a way that leaves no trace in the summary.
None of these are new. What has changed is how confidently they arrive, which is exactly why a plain “looks right” judgment is unreliable.
A scorecard from 60 prompts
Over two weeks I ran 60 prompts lifted from actual work: 22 factual lookups, 18 rewrites and summaries, 12 calculations, 8 code tasks.
Fifteen of the 22 lookups were right first time. Four were partly wrong, usually a correct figure attached to the wrong year. Three were invented outright, and all three involved niche statistics where no clean source exists.
All 18 summaries were faithful, though two dropped a qualifying caveat that changed the meaning. Nine of the 12 calculations were correct; the three failures were all unit conversions. Seven of the eight code snippets ran, and the one that didn’t called a library function deprecated a year earlier.
Call it 80% reliable on first pass and 95% after a single check. That is a genuinely useful tool and a genuinely bad oracle.
Plus versus free, on accuracy alone
The free tier now runs on the same model family, so for a one-off question the gap is smaller than the price suggests. The difference shows up in volume and in task length: message limits, file uploads, web access, memory, and the slower reasoning modes. Accuracy on a short question is close to a draw. Accuracy on a two-hour research task is not close at all.
Subscriber behaviour reflects that split. Reviews of what people praise and quietly cancel over tend to praise the long tasks and cancel over the monthly cost of work they could have done in a single session.
How it compares with the alternatives
On factual recall with citations, tools built around search still beat it. On reasoning over documents you hand it, it sits at or near the front of the pack. A broader accuracy comparison across paid AI tools lands in roughly the same place, and the advice holds: pick by task type, not by leaderboard position.
Habits that cut the error rate
- Ask for the source, then open it. If the link doesn’t resolve, treat the claim as fiction.
- Paste the document instead of asking it to remember the document.
- Ask what would change the answer. A vague reply is a signal.
- Rephrase and ask again. Facts that flip between phrasings are facts to check.
- Keep a short list of things it has got wrong for you before. The patterns repeat.
Where it pays off in a business setting
Drafting, rewriting, first-pass research, code review and meeting summaries are the reliable zones. Anything customer-facing needs a human read before it ships, and anything numeric needs a second calculator. The value-for-money question for business tools comes down to the same test as here: how long does verification take compared with doing the work directly?
The question that decides whether you renew
Accuracy reviews chase a single score. The useful number is personal: how many minutes does it take you to check an answer, and how often does that check need to happen?
If you can verify a claim in under a minute, Plus is close to free. If verifying takes longer than writing the thing yourself, the subscription is buying you anxiety. And if the answer will be published, forwarded or acted on without a second reader, no assistant’s confidence is worth much. Mine got a number wrong by a factor of two and I only found out from a reader. The model has improved since. The lesson has not.