It’s 9:12 on a Sunday night and someone’s boiler has packed in. They ring the number on the website expecting voicemail. A voice answers instead, asks what’s wrong, offers an 8am slot on Monday, takes a mobile number and reads the postcode back to check it. The engineer sees the job on her phone before she’s had her coffee.
That’s an AI voice agent: software that listens to speech, works out what the caller actually wants, and replies out loud in something close to real time. No button pressing, no “press 1 for sales”.
A lot of businesses are now running these systems on live numbers and most callers never twig. The gap between a good one and a bad one is huge, and it usually shows up in the first fifteen seconds.
What separates a voice agent from the phone menu you already hate
Traditional IVR runs on touch tones. It’s cheap, it never crashes, and callers loathe it. A voice agent swaps the menu for conversation, which means three components have to work together: speech recognition to turn audio into words, a language model to infer intent and pick a reply, and speech synthesis to say it back in a voice that doesn’t sound like a 1990s satnav.
Then there’s the part vendors tend to undersell: latency. People tolerate roughly a second of silence on a call. Push past two seconds and they start saying “hello? are you still there?” So the real engineering is less about the model and more about shaving milliseconds off the round trip.
Decent systems also handle interruption. Cut in halfway through the agent’s sentence and it should stop talking and listen. The bad ones plough on to the end of their script while you talk over them, and that single flaw makes the whole thing feel like a machine reading a card.
Voice is unforgiving in a way chat isn’t. There’s no scrollback and no edit button. If the agent mishears your email address you can’t see what it typed, which is why the good ones confirm the details that matter: phone numbers, dates, awkward spellings.
The work they handle well right now
Narrow, well-defined jobs with a clear finish line. That’s the sweet spot.
- After-hours booking. Trades, dental practices, vet clinics, salons. The calls that used to hit voicemail and die there.
- Overflow. Lunchtime at a takeaway, Monday morning at a letting agency. The agent absorbs the queue instead of letting it ring out.
- Rescheduling and cancellations. Low emotional stakes, high volume, tedious for staff.
- Lead qualification. A few screening questions, then a booked slot or a warm transfer to whoever sells.
- Outbound confirmations and reminders. Unglamorous, quietly effective at cutting no-shows.
The arithmetic behind a missed call
Take a heating firm that misses forty calls a week. Average job value sits around £250 and roughly a quarter of enquiries convert, so £2,500 of work walks away every seven days. If a voice agent captures even 40% of those calls, the conversation stops being about technology and starts being about payroll. After-hours cover is where nearly everyone starts for exactly this reason.
Where voice agents come unstuck
The demo call is always clean. Quiet room, neutral accent, caller who knows what they want. Real calls are nothing like that.
Background noise wrecks recognition accuracy. A van engine, a barking dog, three people in a kitchen. Heavy regional accents and callers who switch languages mid-sentence push word error rates up sharply. So does anything needing spelling: postcodes, email addresses, surnames nobody can pronounce.
Multi-party calls are worse. A couple disagreeing about whether the appointment is Tuesday or Thursday, both talking at once, one of them grabbing the handset. The agent gets two streams of speech and picks the wrong one.
Emotion is the other fault line. Someone ringing about a burst pipe at midnight, or a bereavement, or a refund they’ve chased four times. A calm, evenly modulated synthetic voice lands badly with a person who is furious. It doesn’t read as empathy. It reads as being handled.
Anything requiring judgement, a policy exception, or an apology with actual authority should go to a human. That’s not a temporary limitation, it’s a design boundary.
The handoff is where deployments live or die
A caller asks for a human. What happens in the next ten seconds decides whether they stay a customer. Good version: the agent says “putting you through to Priya, she’ll have the notes”, the phone rings once, and Priya opens a summary with the name, number and problem already on screen. Bad version: hold music, a queue, then “can you give me your details again?”
Warm transfers need the agent and the phone system to talk to each other properly. Skipping that integration is the most common reason a pilot gets written off as a failure. The pattern-matching limits show up in AI customer service agents too, and a clean escalation path is usually the difference between a useful tool and a complaint thread.
The numbers that tell you whether it’s working
Vendors quote containment rates the way bars quote happy hour. Ask for these instead, measured on your own traffic.
- Task completion. Did the booking land in the calendar, or did the agent merely say it would?
- Transfer rate and reason. A 30% transfer rate is fine if those are genuine escalations rather than confusion.
- Repeat calls within seven days. The clearest signal that something was promised and not delivered.
- Handle time against your human baseline, not against the vendor’s demo script.
- Word error rate on recordings of your actual callers, segmented by accent if the tooling allows it.
- Cost per resolved call. A cheap agent that books nothing is the expensive option.
Piloting one without poisoning your customer base
Start narrow. One intent, one time window, after-hours only for the first month. Give it the easiest 20% of your call volume and let it earn the rest.
Disclose that it’s automated. In several US states, including California and Illinois, all-party consent is required to record a call, and the UK and EU bring their own rules under GDPR. Get that line right at the very start, before anything is captured.
Build an escape hatch that always works. “Human”, “agent”, “person”, “representative” should each route you to a real one first time, with no argument and no second attempt at selling you something.
Test on your own recordings before you test on customers. Ten archived calls from your worst-signal time of day will teach you more than a month of vendor sandbox demos. Then read transcripts weekly for the first six weeks. The failures cluster fast and usually come down to five or six phrases the agent keeps mangling.
Plan for the fact that the work doesn’t vanish, it relocates. Someone still has to review transcripts, untangle double bookings and chase the callbacks the agent promised. That’s the same trap described in AI task automation that quietly moves the work around rather than removing it.
Two integration decisions matter more than the voice itself. If your booking system has no API, you may need software that drives the interface directly, which is where AI computer use agents that click and type through legacy systems become relevant. And the confirmation after the call is not an afterthought: a solid post-call email automation flow catches the half of callers who weren’t listening properly when the slot was read out.
What’s shifting over the next year
The three-stage pipeline is being replaced by speech-to-speech models that process audio in and audio out, cutting latency and preserving tone in ways the stitched version can’t. Sub-500ms responses are already shipping. Agents are also starting to act mid-call rather than just talk: take a payment, check stock, pull up a record, reschedule in the live system.
The consumer side is moving too. Voice assistants on phones are beginning to place calls on your behalf, a capability examined in what an AI personal assistant really does in 2025. When both ends of a phone call are software, the interesting questions stop being technical and become about disclosure, consent and whether anyone should be allowed to pass a robot off as a receptionist.
Six questions to ask before you sign anything
Bring these to the demo call and watch how the sales engineer reacts. Hesitation tells you plenty.
- Can I hear a recording of a real customer call in my industry, not your reference script?
- What exactly happens when the caller is angry, crying or threatening to cancel?
- Does the human receiving a transfer see a transcript and the caller’s details, or just a ringing line?
- What’s the measured word error rate on accents like the ones in my postcode area?
- Who owns the call recordings, where are they stored, how long are they kept, and how do I delete them?
- What’s the all-in cost per minute including telephony, model tokens and support, and what’s the contract minimum?
If a vendor can answer all six without flinching, you’re probably talking to someone who has deployed this for real. If they pivot to a slide about digital transformation, book the next demo.