The last three voice agent pilots I sat in on failed for the same unglamorous reason. None of them broke because the speech recognition was weak or the voices sounded robotic. They broke because the team pointed the thing at the hardest calls in the queue, skipped the prep work nobody enjoys, and never agreed on what a good result looked like. Poly AI handles a serious share of phone volume once it is configured properly. Configuration is the part that decides whether you get a useful deployment or an expensive experiment.
If you want the market-level view of what the platform is and why contact centres care, there is a solid overview of how Poly AI is reshaping customer service. What follows is the build guide: six weeks, in order, with the decisions that actually trip teams up.
Step 1: Pick a call type that pays for itself
Resist the temptation to automate everything at once. Choose one call type that passes four tests:
- Volume. At least 2,000 calls a month, so your data means something and your savings are visible.
- Low variance. Ten possible answers, not ten thousand.
- Low emotional stakes. “Where is my parcel?” works. “Why was my late father’s account closed?” does not.
- An existing process. If your human agents improvise every call, there is nothing to teach the agent.
Order status, appointment rescheduling, delivery delay notices, store stock checks and password resets all qualify. Run the arithmetic before you commit. A retailer taking 40,000 calls a month where 18% are order status has 7,200 calls in scope. Contain 65% of those and you have removed roughly 4,700 calls from the queue every month. Even at a modest cost per contact, that pays for a pilot quickly, and it is the kind of number that survives a finance review.
Step 2: Map intents from real transcripts
Do not build the intent list from a workshop whiteboard. Pull 400 recent call recordings from the call type you chose and tag them by hand. It takes two people about three days. What you find will change the design.
The pattern that shows up almost every time is the second question. A caller asks for their balance, then spends four minutes on a disputed charge. A caller asks about a delivery date, then reveals they have moved house and the address is wrong. Your intent map has to cover the second question, or the agent will try to end a call the customer is not finished with.
Build a table with four columns: intent, common trigger phrases, the data the agent needs, and a one-line definition of success. “Reschedule delivery” succeeds when a new date is confirmed and the customer hangs up without asking for a human. That last column is the one teams forget, and it is the one you will measure against later.
Step 3: Write the hard-stop list before the happy path
Before you design a single conversation flow, write down what the agent must never do. Most deployments draw this line too late.
Typical hard stops include refunds above a set value, anything involving medical or legal advice, complaints naming a specific employee, threats of cancellation on accounts over a certain size, and any caller who asks for a human twice. Authentication is a separate decision: what data is the agent allowed to read out, and what identity checks happen before it does? Date of birth plus postcode is weak. A one-time code sent to the registered number is stronger and adds about eleven seconds to the call.
Then design the handoff itself. A warm transfer that passes the transcript, the verified identity and the reason for the call cuts handling time on the human side. A cold transfer that drops the caller back into the IVR queue undoes everything the agent just achieved.
Step 4: Wire the data before the dialogue
Voice agents that sound impressive in a demo and fail in production almost always fail on data. If the agent cannot see the order management system, it will guess, and guessing to a customer about their money or their delivery is the fastest way to lose trust. Get the API connections working first, then write the conversation.
Latency deserves a budget too. Aim to keep the gap between the caller finishing a sentence and the agent responding under 800 milliseconds. Anything past 1.2 seconds and callers start talking over the agent, which triggers a cascade of interruption handling problems.
This is also the point where you decide whether to buy or build. If your engineering team is pushing to assemble something in-house on a general model, read the breakdown of what the OpenAI API costs and what you can realistically build with it before anyone commits to a timeline. Speech-to-text, telephony, barge-in handling and analytics are four separate workstreams, and only one of them is the fun part.
Step 5: Test with recorded audio, not written scripts
A script on paper tells you nothing about how a voice model copes with a caller who is driving, eating, or standing next to a washing machine. Replay real recordings through the agent and score each one.
Build a trap list of genuinely awkward calls: heavy regional accents, people who spell their postcode letter by letter, callers who interrupt mid-sentence, someone who changes the subject three times, and at least a dozen recordings where the customer is already annoyed before the agent speaks. Twenty of these will surface more bugs than two hundred clean test calls.
Run the same 200-call suite after every configuration change. It is the only reliable way to know whether a fix broke something you shipped last week.
Step 6: Canary at 10% and watch three numbers
Send a tenth of live traffic to the agent for two weeks. Containment rate gets all the attention, but it is the least honest metric on its own; an agent can contain a call by frustrating someone into hanging up. Track three together:
- Containment rate, so you know how much volume left the queue.
- Repeat calls within 24 hours, the hidden failure signal. If it climbs, the agent is closing conversations without solving anything.
- Satisfaction on contained calls only, measured separately from human-handled calls.
It also helps to know what the rest of the field is shipping, partly so you can benchmark, partly because procurement will ask why you did not build this yourself. The 2025 ranking of the leading AI companies is a reasonable starting point when that question arrives.
Where deployments break, and the fixes
Four problems account for most of the trouble I have seen. Accents and proper nouns get mangled, which is fixed by adding phonetic entries for place names and brand names rather than retraining anything. Callers switch topics mid-call, which is fixed by keeping the last three intents in context instead of resetting after each turn. Background noise triggers false barge-ins, which is fixed by raising the voice activity threshold slightly and accepting a fraction of a second more latency.
The fourth is regulatory. Calls that touch financial advice, insurance eligibility or medical information carry obligations that a general-purpose voice agent should not be asked to shoulder. Teams in banking learned this lesson early on the trading side, where the same standards of explainability apply to machine learning models making decisions about people’s money. Voice is a newer channel, but the scrutiny is identical.
Week seven and beyond: the tuning that moves numbers
The work after launch is quieter and more valuable than the work before it. Set a weekly routine and keep it short.
Review the twenty longest contained calls and the twenty most common transfer reasons. Delete intents that fire fewer than five times a week; each dead intent adds a branch the model has to consider. Add phrasing variants pulled straight from transcripts rather than invented ones. Listen for the point where callers repeat themselves, because that repetition marks the exact sentence the agent misunderstood.
Most teams find that containment climbs five to ten points over the first two months without any change to the underlying model, purely from pruning and phrasing work. Set a target, say 70% containment with repeat calls flat, and hold the agent to it before widening its scope. Then pick your next call type and start again, because the second deployment takes about half as long as the first.