Take a company with 14 employees and 260 support tickets a day. Two people spend their mornings sorting refund requests into three piles: approve, ask for a photo, escalate to a human. It is dull, repetitive work, and it leaves a paper trail. That is exactly the kind of job an AI agent can absorb in a fortnight.
Most agent projects that stall do not stall because the model is not clever enough. They stall because nobody decided what the agent was allowed to touch, how anyone would know it was doing a good job, or when to take the keys back. That is a deployment problem, and deployment problems have a playbook.
Here it is, in seven steps. If you are still choosing a framework and wiring up your first tool call, start with a step-by-step guide to building your first AI agent, then come back here once it actually runs.
Score the job before you automate it
Not every task deserves an agent. The ones that do share three traits, so run your candidate through this list:
- It happens at least 20 times a week. Below that, the setup cost never pays back.
- A new hire would need most of a day to learn it. If a two-line rule covers the whole thing, write a script instead.
- When the agent gets it wrong, a person can still catch it before money leaves the building or data leaves the company.
Refund triage clears all three. Choosing which influencer to sponsor fails the third one badly, so it stays with a human. A task that fails only the third test is still worth building, just build it read-only first: it recommends, a person presses the button.
Write the job down like a checklist for a temp
The most valuable hour you will spend on this project is the one where you type the rules out by hand. For refund triage, the first draft looked like this:
Approve refunds under 40 pounds without asking. Between 40 and 150, check the order is inside 30 days and the customer has not had a refund in the last 90. Above 150, escalate to a human. If the customer mentions a chargeback, escalate immediately and tag it legal-risk. Never promise a delivery date.
That paragraph effectively is the agent. The model is the engine that reads it. Where people go wrong is writing the happy path and stopping there, because the edge cases are the job. Spend a second hour pulling 30 real tickets and writing down the correct answer for each. You now have a test set, which will matter far more than it seems right now.
Give it tools, not a chat window
A chatbot that can only talk is a demo. An agent earns its keep when it can look something up and change something. Three tools is a sensible place to start:
- lookup_order(email) reads order history. No writes.
- issue_refund(order_id, amount) writes, but with a hard cap.
- tag_ticket(ticket_id, label) writes a label and nothing else.
Every extra tool is another way for the agent to surprise you at 2am. Start with the smallest set that lets it finish one job end to end.
Decide what it remembers, and where that memory lives
An agent that forgets the customer complained twice last month will cheerfully approve the third refund and look foolish doing it. Memory is what turns a clever autocomplete into something that behaves like a colleague. Open-source options exist: Letta gives an agent a persistent memory layer you can read and edit, rather than a black box you have to trust.
Keep that memory scoped, though. The support agent’s notes on a customer should not bleed into the marketing team’s agent, and neither of them needs to hold card details just because it saw them once. If a full memory layer feels heavy for version one, a 20-line summary of the customer passed into every prompt gets you most of the benefit for a fraction of the complexity.
Lock the blast radius before it goes live
Give the agent its own credentials. Nothing shared with production dashboards, nothing with admin scope, and rotate the keys on a schedule. Allowlist the domains it can reach. Set a daily spend ceiling inside your model provider’s dashboard, low enough that a runaway loop is annoying rather than expensive.
Keep the first version on draft-not-send for anything outbound, too. An agent that writes the email while a human clicks send has a completely different risk profile from one with a live send button. There are already cautionary tales about unsecured agents publishing user data without anyone noticing. The fix is boring permissions, applied before launch rather than the morning after.
Run it in shadow mode for two weeks
Do not hand over the inbox on day one. For two weeks, let the agent read every ticket and produce a proposed decision while a human makes the real call and the agent’s output stays invisible. Then compare. Three numbers tell you nearly everything:
- Agreement rate: how often the agent matched the human.
- Escalation rate: how often it asked for help. Too low is more worrying than too high.
- Cost per ticket: model spend, tool calls and review time, added together.
A store running this on refund triage might see agreement climb from about 71% in week one to roughly 84% in week two, usually because someone tightened a threshold rule rather than swapped models. If agreement is still under 80% after a fortnight, the spec is almost certainly the problem. Rewrite the rule before you buy a bigger model.
Widen autonomy one notch at a time
Autonomy is a dial, not a switch. Turn it one notch and hold it there:
- Drafts only. The human sends everything.
- Low stakes only. It acts on reversible, small-value cases, such as refunds under 40 pounds.
- Mid stakes, with notification. It acts, then tells a human what it did.
- Exception handling. It acts on everything inside spec and escalates only what falls outside.
- Self-tuning, within limits. It proposes changes to its own thresholds and you approve them weekly.
Never skip a notch. Each one should hold for two weeks without a nasty surprise before you turn the dial again, and there should be one obvious way to switch the whole thing off. Name that switch in the spec so nobody has to guess at 3am.
Keep a human on the exception path, and keep the receipts
The point of an agent is not to remove people from the loop. It is to move them to the part of the loop that needs judgement. That makes the escalation path a feature, not a failure. An agent that escalates 11% of tickets and gets those right beats one that never escalates and quietly approves the wrong 3%.
Log everything: the input, the tool calls, the output, and the rule the agent claims it followed. When it does something strange, you need to know whether it misread the spec or ignored it, and that distinction matters more when money is involved. Questions about who is liable when AI agents go rogue are still unsettled in most jurisdictions, and your logs are the first thing anyone will ask to see.
If there is no API, teach the agent to browse
A lot of the work worth automating sits behind a supplier portal that never got an API. Browser agents close that gap: they log in, click through a dashboard, download the invoice and drop the numbers into your accounting tool. The toolkit for agents that click, type and browse on your behalf has matured quickly over the past year.
Treat browsing as a bridge rather than a destination. It is slower than an API call, it breaks whenever a page gets redesigned, and it needs more careful credential handling because the agent is driving a real session. The moment a proper API appears, migrate to it and retire the clicking version.
What a healthy agent looks like after 90 days
Come back three months in and the shape of a good deployment is fairly consistent. Escalations fall from roughly 38% of tickets to around 11%, not because the agent got braver but because the spec got sharper. Cost per ticket drops from a few dollars to under 50 cents. Two people who used to sort refunds now handle the exception queue and use the time they got back for something better.
You will also have rewritten the spec eight or nine times, which is normal and healthy. The document is the product. And resist the urge to launch three agents at once; pick one job, ship it small, watch it for a month. The second agent will be easier than the first, because by then you already own the rules, the tool definitions and the permission scaffolding, and none of that needs to be invented twice.