A mid-sized ecommerce team hired an artificial intelligence agency last spring to build a support chatbot. Three months and $48,000 later, the bot was closing 6% of tickets end to end, and the support reps had quietly gone back to copy-pasting macros. The model was fine. The brief was not.
Nearly every stalled AI project I’ve seen follows that shape: a vague ambition, a polished demo, and no agreed way to tell whether the thing actually worked. The fix is mostly procedural. Here’s the sequence I’d run before, during, and after hiring a firm, with the documents and numbers you need at each stage.
Step 1: Write a job description for the AI
“We want an AI assistant” gives an agency nothing to work with. Compare it with this:
Draft first responses to inbound insurance claims, in our tone, and drop them into an adjuster’s queue within 90 seconds of the email arriving. Four adjusters currently spend about 14 hours a week on first drafts.
The second version names the input, the output, the latency, the volume, and the human’s role. Now someone can scope it, price it, and test it.
Add what a bad output costs. A wrong delivery estimate in a support reply is annoying. A wrong clause in a contract summary is legal exposure. Agencies staff and design very differently depending on that distinction, so make it explicit instead of leaving them to guess.
It also helps to know what you’re shopping for. Firms range from strategy shops that deliver slide decks to engineering teams that ship production systems, and what an artificial intelligence agency actually does varies enough that the same brief can come back at $8,000 or $80,000.
Step 2: Baseline the work as humans do it
You can’t tell whether an AI is winning if nobody has measured the person doing the job today. Take a week and count things.
A 12-person law firm I worked with tracked 340 contract reviews a quarter, 45 minutes each, with roughly 3% escalating for unusual liability clauses. Those three numbers became the scoreboard. Six weeks later, when a vendor claimed their tool “saved 60% of review time,” the firm could check the claim: was that 27 minutes saved on standard contracts, or was the tool fast on easy work and useless on the strange ones?
If your team can’t describe the current process in plain steps, no model can slot into it. This is where simple rules-based automation often earns its keep first, and it’s worth understanding the difference between a scripted workflow and a reasoning agent before a vendor blurs the two — what intelligent agents are and why they matter is a decent starting point.
Step 3: Design a 30-day proof with a kill switch
Skip the 12-month roadmap. Pick one workflow, one team, one metric, one deadline. I default to a paid 30-day pilot: long enough to surface the ugly edge cases, short enough that nobody’s career depends on the result.
What goes in the pilot charter
- Scope: one queue, one document type, one region. Not “all customer interactions.”
- Metric: something countable — average handle time, first-response time, or the percentage of items needing human correction.
- Target: a number agreed in advance. Cutting draft time from 45 to 20 minutes is a target. “Improving efficiency” is not.
- Kill criteria: the result that stops the project. Decide it while everyone is still optimistic.
- Gating item: whatever the agency needs from you in week one — system access, historical tickets, two hours a week of a subject-matter expert.
That last bullet kills more pilots than anything technical. If CRM access takes six weeks to arrange, your 30-day pilot is really a five-day trial with a long wait attached.
Step 4: Demand an evaluation set in week one
The clearest signal of a serious team is that they build the test before the product. Ask for 50 real examples from your own history, scored by your own staff as pass or fail, with the disagreements written down.
That set does three jobs. It shows whether their model handles your actual data rather than a curated sample, it gives everyone a shared definition of “good,” and it becomes the thing you re-run every time someone touches a prompt. Without it, you’re grading a demo, and demos are graded on vibes.
When you’re comparing firms, watch how each one reacts to this request. It sorts the shops that can deliver from the ones that can only present, which is the entire exercise in how to choose an AI agency that actually delivers results.
Step 5: Launch with a human in the loop, then widen it
Weeks one and two: the system drafts, a human approves everything, and you log every edit. Those edits are free training data, and the approval timestamps tell you whether you’ve actually saved time or just moved the work.
Weeks three and four: switch on auto-send for the narrow categories that scored cleanly, say order status and appointment confirmations, and keep the rest in review.
Keep a miss log. Every override gets one line explaining why. After a month you’ll have a list of edge cases that no amount of prompt engineering in a conference room would have predicted. It’s the same logic behind automating your workflow with intelligent agents: prove a narrow job, then expand the perimeter.
Step 6: Price it per task, and plan the day you turn it off
Monthly seat licenses hide the number that matters. Cost per completed task is the one to track. If a drafted claim response costs $0.14 in compute and tooling against $4.20 of adjuster time, you know exactly where the line sits, and you’ll know the moment a pricing change breaks the maths.
Ask for three things at the end of any pilot, whether or not you continue:
- Your data back in a usable format, plus the evaluation set you built together.
- Documentation of prompts, model versions, and integrations — enough that another team could pick it up.
- A handover session with the people who will run it daily, not just the executives who signed the contract.
Systems drift. Product lines change, regulations shift, and a workflow routing 92% of tickets correctly in March might be at 71% by September. Schedule a monthly re-run of the evaluation set and give one named person the number.
If the project touches cameras, add a governance step
Vision work looks deceptively simple in a pilot: count people in a queue, spot empty shelves, check for safety helmets. The technical side is often the easy part. The hard part arrives when the system is wrong in public.
Camera-based deployments attract scrutiny that text tools rarely do. The death of a woman at the San Diego border, in a place where a surveillance camera sat in plain sight, is a reminder that “the footage existed” and “someone acted on it” are two entirely different claims. Before approving any camera pilot, write down who reviews false positives, how long footage is retained, and what the failure looks like for the person on the other end of the lens.
None of this is glamorous. It’s a stopwatch, a one-page brief, 50 scored examples, and a kill criterion nobody enjoys writing. That’s the work that separates a pilot that ships from the $48,000 chatbot nobody mentions at the next quarterly review.