An agent that completes a task eight times out of ten feels like a win in the demo. In production, it means one in five customers hits a dead end, and nobody knows which one until the support tickets arrive.
That gap is what AI agent testing exists to close. Not by proving the agent is perfect, but by mapping exactly where it breaks, how often, and whether the break is annoying or expensive.
Most teams skip this. They run a handful of prompts in a notebook, see good outputs, and ship. Two weeks later they’re reading transcripts at midnight trying to work out why the refund agent started promising customers store credit it has no authority to grant.
Why agent testing breaks the tools you already use
Unit tests assume determinism. Give the function the same input and you get the same output, forever. Agents don’t work that way. The model samples, the tools return different data on different days, and a conversation that started about invoices can wander into territory you never anticipated.
Three things make agents genuinely awkward to test:
- Non-deterministic outputs. The same prompt can produce three different but equally correct plans. String matching gets you nowhere.
- Compositional behaviour. Failures often live in the joins between steps, not inside any single step. Each tool call looks fine; the sequence is wrong.
- State that accumulates. Turn twelve depends on turns one through eleven. A bug introduced early may only surface much later.
The practical answer is to assert on outcomes and constraints rather than exact text. Did it achieve the goal? Did it stay inside the boundaries? Did it cost less than 4 cents and finish in under 12 seconds? Those are testable. The precise wording it used to say hello is not.
Write a behaviour spec before you write a test
Teams that jump straight into test cases usually end up with a scattered pile of prompts that covers the happy path and nothing else. Spend an hour first listing what the agent is supposed to do, in plain language, with pass criteria attached.
For a customer support agent, that might read: process refunds up to $500 without approval; escalate anything above $500 to a human queue; never discuss competitor pricing; never send email to an external domain without confirmation.
Separate invariants from quality bars
Some rules are absolute. The agent must never issue a refund twice for the same order. Break that and you have a financial incident. Other rules are matters of degree: tone, concision, helpfulness. Mixing the two in one test suite means a slightly snarky reply fails the build the same way a duplicate refund does.
Keep hard constraints as binary gates. Score the softer stuff on a 1 to 5 rubric and track the average over time. That way a small drift in tone shows up as a trend, not a false alarm.
Test each layer on its own
Agents fail at different altitudes, and a single end-to-end suite tells you something broke without telling you where. Split the testing into layers.
Tool contracts and function calling
Before worrying about reasoning, confirm the agent can reliably pick the right tool and fill in valid arguments. Malformed parameters, wrong date formats, and hallucinated field names are the most common early failures. Understanding how function calling actually works under the hood makes these tests far easier to write, because you know what the model is really being asked to produce.
Single-turn reasoning
One user message, one expected decision. Should this ticket be routed to billing or to technical support? Should this discount request be approved? These run fast and cheap, so you can afford hundreds of them in every build.
Multi-turn trajectories
Here you check the whole path: which tools were called, in what order, with what arguments, and whether the final state of the system is correct. A good habit is to snapshot the full trace and diff it against an approved run. You’re not looking for identical steps, just for the same destination reached without forbidden moves.
Multi-agent handoffs
Once you add a planner, a researcher, and a writer that pass work between them, the failure surface multiplies. Handoffs drop context, agents disagree about who owns a task, and loops appear out of nowhere. If this is your architecture, the testing patterns in this guide to orchestrating multiple agents so they behave like a team are worth reading alongside this one, because the observability requirements are different.
Build a golden set from real traffic
Synthetic test cases are fine for day one and misleading by week three. Nothing predicts production like production.
Pull a few hundred real transcripts from your logs. Label the expected outcome for each. Aim for 200 to 500 cases to start, weighted toward the messy ones: the customer who changed their mind halfway through, the order that exists in two systems, the request with a typo in every second word.
Two rules that save pain later. First, include the boring cases. Agents often fail on trivial inputs because they over-think them. Second, keep a held-out slice of about 20 percent that you never look at while iterating. It’s the only honest measure you’ll have of whether you’ve overfitted to your own test set.
Grading without a human on every run
You cannot read 500 transcripts after every commit. Automate the grading, then verify the grader.
Start with deterministic checks wherever possible. Did the agent call the correct tool? Did the arguments validate against the schema? Did the database end up in the right state? These are cheap, fast, and unambiguous.
For everything else, an LLM judge with a written rubric works well, provided you calibrate it. Have a human label 50 cases, run the judge on the same 50, and measure agreement. Below roughly 80 percent agreement, tighten the rubric before you trust any number the judge produces.
Failure modes that show up after launch
These are the ones that slip past clean demo runs and surface in week two:
- False success. A tool returns a 500 error and the agent cheerfully tells the user the task is complete.
- Goal drift. Over a long conversation the agent slowly abandons the original request in favour of whatever was mentioned last.
- Injection through retrieved content. A support document or web page contains instructions that hijack the agent’s behaviour.
- Retry spirals. The agent retries a failing call nine times, burning tokens and hitting rate limits.
- Memory bleed. Context from one user’s session leaks into another because the session key was set incorrectly.
Write a test for each one. They’re cheap to build and they catch the failures that generate the angriest customers.
Run evals like a CI pipeline, not a launch event
Testing an agent once, before launch, is the same as testing a web app once, before launch. The interesting failures appear after the fifth prompt change and the model upgrade nobody announced.
Wire your eval suite into CI. Run the fast layers on every pull request, the full suite nightly, and block deploys when task success drops more than two points or cost per task climbs above your ceiling. Keep a dashboard with the numbers that matter: success rate by intent, escalation rate, mean tool calls per task, p95 latency, and cost per completed task.
This is roughly the line described in what separates a working demo from a system you actually trust, and it holds whether you built the agent from scratch or assembled it in a low-code tool. If you went the low-code route, be aware that low-code agent builders ship fast but often break at scale in ways your test suite needs to anticipate, particularly around concurrency and rate limits.
Start with the failures you already know about
You don’t need a perfect test framework to begin. Open last week’s transcripts and find twenty conversations where the agent did something wrong. Twenty is enough. Write each one as a test case with the expected outcome spelled out, run them, and watch your pass rate.
That number is your real starting point, not the demo. From there, add a case every time something breaks in production, and within a month you’ll have a suite that reflects how your agent actually behaves rather than how you hoped it would. The teams that get reliable agents aren’t the ones with the cleverest prompts. They’re the ones who kept a record of every way it failed and refused to let it fail the same way twice.