Most conversational AI projects die the same way. Someone buys a platform, points it at every question a customer could ever ask, and waits. Three months later it’s answering “I’m sorry, I didn’t understand that” to a billing question and the team quietly switches it off.
The teams getting real numbers out of this stuff do the opposite. They pick one narrow job, write the dialogue by hand before touching a model, and only then wire it up. It’s less exciting than the demos, and it’s the difference between a bot that handles 40% of your support volume and one that generates complaints.
What follows is the sequence I’d run today with a support queue, a modest budget, and eight weeks. Part of the reason this is even feasible now comes down to the reasons conversational AI suddenly got good, but the method matters more than the model.
Step 1: Pick one job with a measurable cost
Open your ticket system and sort by volume, not by how impressive the problem sounds. You’re looking for a request that is high-frequency, low-variance, and already answered from a known source of truth.
A real example: a 40-person ecommerce company found that 38% of its live chat volume was some version of “where is my order?” That’s a single intent with three possible answers pulled straight from the shipping API. It took their team eleven days to automate 80% of those conversations. Choosing “answer all customer questions” instead would have taken a year and still failed.
Schools doing this well narrow it the same way. The projects that work are the ones solving a specific problem, which is a pattern worth studying in AI in education, where narrow tutoring bots beat general assistants every time.
Score candidates against four filters
- Volume: at least 50 a week, or the ROI math never closes.
- Data availability: the answer lives in an API, a database, or a document you can point to.
- Low emotional stakes: password resets, yes. Refund disputes, no.
- Fast feedback: you can tell within a day whether the answer was right.
Step 2: Write the script before you write any code
This is the step everyone skips. Run a wizard-of-Oz week: have a human agent reply to real customers using a script you’ve drafted, with no AI involved. Fifty conversations is enough to expose the branches you didn’t imagine.
Log four things for every intent you find:
- The exact phrases customers use, typos included.
- The data you need to answer (order ID, email, account number).
- The fallback when that data is missing.
- The trigger that hands the conversation to a human.
By the end of that week you’ll have a conversation tree that reflects reality. When you later hand it to an LLM, the model’s job is to route and phrase, not to invent policy. That distinction is what keeps assistants from confidently making things up, a failure mode covered well in what ChatGPT-style assistants get wrong and how to make them useful.
Step 3: Decide between an API, a platform, or a hybrid
Three viable builds, and the right one depends on who maintains it in month six.
Off-the-shelf platform
Fastest to launch, weakest at anything custom. Fine if your use case is genuinely standard and you never need to touch retrieval logic.
Direct API build
Full control, and you own the prompt, the retrieval, and the evaluation loop. Costs scale with tokens, which is usually cheaper than per-seat pricing once you pass a few thousand conversations a month. Run your own numbers against what the OpenAI API actually costs and where it falls short before committing, because token spend on long conversations adds up faster than people expect.
Hybrid
A rules engine handles the deterministic 60% (order lookups, hours, returns policy) and an LLM handles the messy phrasing and summarisation. This is what most production systems actually look like.
Step 4: Ground it in your own documents
Retrieval-augmented generation is the difference between a bot that knows your return policy and one that guesses. The details matter more than the architecture.
Chunk your content by meaning, not by character count. A returns policy split mid-sentence produces garbled answers. Aim for 200 to 400 token chunks with a sensible heading attached, retrieve the top 3 to 5, and instruct the model to answer only from those passages. If nothing relevant comes back, it says so and escalates.
Test retrieval separately from generation. If the right passage isn’t in the top five results, no prompt engineering will save you.
Step 5: Break it on purpose before your customers do
Run at least 200 adversarial conversations before launch. Real ones to include:
- Typed in a hurry at 1am: “wheres my stuff ordered 3 weeks ago???”
- A pasted wall of text containing a hidden instruction to ignore your rules.
- A question in a language you didn’t plan for.
- A genuinely angry customer who wants a manager in the first message.
- Two questions in one message, where only the second one is answerable.
Score each response on correctness, tone, and whether it escalated when it should have. This is where you find the assistants that sound great in a demo and fall apart in production, the gap that deploying an AI bot successfully is really about.
Step 6: Set the handoff rule, then instrument everything
Give customers an exit that works in one tap. A visible “talk to a person” button beats a bot that argues. Then decide the automatic escalation triggers: two failed attempts, any mention of legal or a chargeback, sentiment dropping below your threshold, or any request outside the approved intents.
Track four numbers weekly from day one:
- Containment rate: conversations resolved without a human. 45 to 60% is a realistic target for a single well-scoped intent.
- Escalation accuracy: how often the handoff was actually necessary.
- Cost per resolution: total spend divided by resolved conversations. Compare it to your human average.
- Post-chat CSAT: split by contained and escalated. If contained chats score lower, you’re saving money and losing customers.
Containment alone is a vanity metric. A bot that closes 70% of conversations by frustrating people into leaving looks excellent on a dashboard and terrible in a P&L review six months later.
Step 7: Review transcripts every week and fix the top three failures
Read 30 transcripts a week. Tag each failure as missing data, bad retrieval, wrong tone, or a genuine gap in your script. Fix the three most common. That’s it.
This loop is unglamorous and it’s where most of the improvement comes from. Week one you’ll find twelve problems. By week six you’ll find three, and one of them will be a policy question your own human agents disagree about. At that point you’ve moved past the technology problem and back to the operations problem, which is exactly where a working conversational AI should leave you.