A support queue at 11:40pm on a Sunday has its own particular loneliness. Someone’s washing machine has flooded the kitchen floor, they’ve been through the help centre twice, and the chatbot keeps asking them to describe the issue in a few words.
That gap between what vendors promise and what customers actually get is most of the story of AI customer service agents right now. The technology is genuinely useful. It’s also routinely deployed badly, in ways that turn a five-minute problem into a forty-minute one.
Here’s what these systems do well, where they quietly fall apart, and which numbers tell you whether yours is working.
What an AI customer service agent actually does
Strip the marketing away and most of these systems are three components bolted together: a language model doing the talking, a retrieval layer that pulls the right paragraph out of your help centre or policy docs, and an action layer that can genuinely do something. Look up an order. Issue a refund under £50. Push a delivery date back two days.
That third piece separates an agent from a chatbot. A bot that can only talk always ends the conversation with “let me transfer you.” One that can query the order database and trigger a reship closes the loop.
The model underneath is doing pattern recognition on text, not reasoning in any human sense, which is worth remembering when a system confidently invents a returns window that doesn’t exist. If you want the mechanics, this plain-English guide to how AI actually works covers the foundations.
Where they earn their keep
Roughly a third of ecommerce support contacts are some version of “where is my order.” Password resets, delivery rescheduling, invoice copies and address changes follow close behind. None of those need judgement. They need a lookup and a policy.
The pattern that works looks like this:
- High volume, low ambiguity. Thousands of near-identical requests answered the same way every time.
- Written policy already exists. If the rules live in someone’s head, the agent can’t follow them.
- Reversible actions. Pausing a subscription is easy to undo. Closing an account is not.
- A clear failure signal. The agent knows when it doesn’t know, and says so out loud.
Break any one of those and you get a system that sounds confident while quietly making things worse.
The Klarna correction, and what it taught everyone
Klarna is the most-quoted case study, and it’s worth reading closely because the company changed its story. In February 2024 it announced its AI assistant had handled 2.3 million conversations in its first month, about two-thirds of all customer service chats, equivalent to the work of 700 full-time agents. Average resolution time fell from 11 minutes to under two.
A year later, CEO Sebastian Siemiatkowski admitted the company had pushed cost-cutting too far and was hiring humans back for complex cases and premium customers. Read that as a warning about which metric was being optimised. If leadership measures containment, you get containment: agents that keep conversations alive without solving anything, and customers who give up rather than escalate.
Voice is the harder half
Text forgives latency. Voice does not. Push past about 800 milliseconds of silence and callers start saying “hello?” or talking over the agent, which triggers barge-in handling, which is where a lot of voice deployments fall over.
Speech recognition also struggles with accents, background noise, and the specific way people say postcodes and order numbers. A text agent can score 60% resolution while the voice version of the same system scores 40%. The build is tractable if you scope it tightly, and this six-week plan for launching a voice agent walks through the sequencing.
The handoff is the whole product
The biggest predictor of whether customers trust an AI agent isn’t model quality. It’s what happens the moment it hits its limit.
Good escalation runs on triggers: a second repeat of the same question, sentiment dropping, the word “manager,” an account flagged as high value, a request outside any policy the agent can see. It also passes context. Nothing enrages people faster than explaining the problem to a bot, then explaining it again to a person.
Watch for two failure modes. Missed escalations are the obvious one, where the customer gives up and churns quietly. False escalations are the expensive one, where your agent hands off 45% of conversations it could have handled and you’ve built a very costly router. Deciding how much rope to give a system is really a question about what autonomous AI can and can’t be trusted with, and the honest answer is that autonomy should be a dial set per task, not a switch.
What it really costs to run
Vendor pricing typically lands near a dollar per resolution for established platforms, against several dollars for a human-handled ticket. That spread is why the business case gets signed. It’s also an incomplete number.
Budget for three things the sales deck skips. Content maintenance, because your retrieval layer is only as good as the help centre it reads and stale articles produce confidently wrong answers. Engineering time, for integrations to order management, CRM and refunds tooling. And transcript review, because someone has to read real conversations every week or quality drifts without anyone noticing.
On the platform question, teams either buy a packaged agent or build on a cloud stack. Building gives you control over tone and escalation logic, though it usually means plumbing a model, a vector store and your own APIs together. If that’s your route, knowing your way around Amazon’s AI stack, from Bedrock to the models behind Alexa+, saves a lot of dead ends.
Five numbers that tell you if it’s working
Treat “containment rate” as a vanity metric. It rewards exactly the wrong behaviour. These five are harder to game:
- True resolution rate. Conversations where the customer’s actual problem closed without a human, confirmed by the follow-up survey or the absence of repeat contact.
- Repeat contact within seven days. The cleanest signal that a “resolved” ticket wasn’t.
- CSAT split by path. Compare AI-resolved against escalated. If AI-resolved sits far lower, you’re funnelling your easiest customers into your worst experience.
- Escalation precision. The share of handoffs that genuinely needed a human. Under 50% means the agent gives up too early.
- Cost per resolution, fully loaded. Include the content team and transcript reviews, not just the licence fee.
Where this goes next
The direction of travel is from answering to doing. Agents that file a claim, book a technician slot or process a return end to end are already in production at a handful of large retailers, and the limiting factor is integration work rather than the model itself. Disclosure rules are arriving too: under the EU AI Act, customers have to be told when they’re interacting with a machine, which turns “pretend it’s a person” from an ethics debate into a compliance problem.
Keep an eye on voice latency dropping consistently under 500 milliseconds, roughly the point where phone calls stop feeling awkward, and on agentic tooling that lets these systems take multi-step actions safely. The 2026 AI breakthrough predictions worth taking seriously cover more of that ground. What won’t change is the part that decides success: knowing precisely which conversations your agent should hand over, and making that handover feel like service instead of a queue.