Last spring a 14-person accounting firm spent six weeks evaluating AI assistants. They compared every public benchmark score they could find, built a 22-page spreadsheet, and picked the model with the best numbers. Three months later they quietly cancelled the subscription. The tool that aced the leaderboards kept fumbling the one job they hired it for: sorting 400 messy client expense exports a month without inventing line items that didn’t exist.
That gap — between what a model scores on someone else’s test and what it does on your desk — is exactly why standard benchmarks tell you surprisingly little about whether a tool is worth paying for. The fix isn’t more benchmarks. It’s a small, boring, homemade test you run yourself. Here’s how to build one in an afternoon.
Write down the job before you write the test
Most people start by pasting a random question into a chatbot and judging the answer by feel. That tells you nothing repeatable. Start instead by naming the three to five tasks the AI is supposed to take over. Be painfully literal.
- Draft replies to inbound customer emails about shipping delays
- Summarise a 40-page supplier contract into eight bullet points
- Turn handwritten meeting notes into a task list with owners and dates
- Convert messy CSV exports into a clean monthly reconciliation
If you can’t name the tasks, you’re not ready to test anything. Choosing software before you’ve defined the job is the most common way these projects stall out — the same trap that makes picking an AI platform feel so much harder than it should.
Build a 20-item test set from real work
Twenty tasks is enough. Ten is workable. The rule is that every item must come from your actual pile, not from your imagination.
Pull from the last 30 days
Open your sent folder, your support queue, your project tracker. Grab 20 examples of the work you want automated, including the ugly ones. That contract with the missing annex. That customer who wrote in all caps. That spreadsheet where two columns are swapped.
Include the exact inputs the tool will see
If the AI will receive scanned PDFs, feed it scanned PDFs. If it will receive a 900-word email thread with five quoted replies, don’t trim it down to two sentences because it looks tidier. Every shortcut you take here inflates your score and deflates your trust later.
Write the right answer down first
For each of the 20 items, note what a competent human would produce. For a shipping-delay reply, that might be: acknowledges the delay, gives a new date, offers one remedy, no refund promises. Now you have a scorecard instead of a vibe.
Score with a rubric, not a gut feeling
“That looks pretty good” is not data. Use four numbers per item:
- 0 — wrong or unusable
- 1 — usable only with heavy editing
- 2 — needs a light edit
- 3 — send it as-is
Add one binary flag: did it invent anything? A model that scores 2.8 but fabricates a shipping date twice out of twenty is more dangerous than one scoring 2.1 that never makes things up. Fabrication in a customer reply costs you a real apology. Fabrication in a contract summary costs you a lawyer.
Five things to test besides accuracy
Accuracy is the headline, but it’s rarely what kills a rollout.
- Consistency. Run item #7 three times. If the answers differ wildly, you can’t build a process around it.
- Instruction-following. Ask for exactly six bullets under 15 words each. Count them.
- Latency. A 40-second wait is fine for contract summaries and unbearable in a live chat window.
- Cost at volume. Multiply the per-task cost by your realistic monthly volume. A tool that looks cheap per prompt can cost £900 a month at scale.
- Recovery from bad input. Paste in a corrupted file or an empty field and see whether it flags the problem or barrels ahead.
If you’re testing coding assistants, add one more: give it a function from your own codebase and check whether it respects your existing patterns or rewrites everything in its own style. That single test explains most of what developers love and hate about tools like GitHub Copilot in practice.
Try to break it on purpose
Once the polite tests pass, get adversarial. Ask the assistant to summarise a contract that contains a clause contradicting an earlier one. Send a ticket written in a mix of English and Portuguese. Request a refund policy for a product you don’t sell and see whether it invents one or says it doesn’t know.
Refusals matter here. You want a tool that says “I can’t find that in the document” over one that produces a confident, plausible, wrong answer. Write down which one you got.
A worked example: the reconciliation test
Say you want to test an AI on expense reconciliation. Build 20 exports from past months, each with a known correct total. Feed them in one at a time and record:
- Final total accuracy (did the number match your manual figure?)
- Number of hallucinated line items
- Time to produce output
- Cost per file at your subscription tier
Two tools might both hit 19 out of 20 correct totals. But if one takes four minutes per file and the other takes 20 seconds, the slower one is a non-starter no matter how elegant its output looks. That’s the kind of finding only your own test surfaces.
Free tiers are genuinely useful for this stage — you can run a 20-item test set on several tools for nothing before committing budget. Just check the usage caps first, since a free plan that throttles you after eight prompts will skew your timing results. There’s a decent rundown of what the no-cost options actually allow in this guide to what you get from free AI chat tools.
Set a pass mark before you see the results
Decide in advance what counts as good enough. Something like: average score of 2.2 or higher, zero fabrications across all 20 items, and under 30 seconds per task. Write it down before you run the test. Otherwise you’ll move the goalposts to fit whichever tool you already wanted.
If nothing clears the bar, that’s a useful answer too — it means the task isn’t ready for automation yet, or needs a narrower scope. Shrink the job and retest. Once something does pass, the next step is wiring it into a real workflow, which is where a structured rollout plan helps; this step-by-step guide to putting an AI platform to work covers the deployment side well.
Re-run the test every 90 days
Models change. A tool that scored 2.4 in January might hit 2.7 in April, or quietly get worse as the provider optimises for cost. Keep your 20 items in a shared folder with the original answers and the score sheet. Re-running takes 40 minutes and gives you a defensible reason to keep, switch, or cancel.
That folder is the real asset here — more valuable than any leaderboard, because it measures the only thing that matters: whether this particular tool does your particular work, at a price and speed you can live with.