The demo is not the evaluation
A support team I worked with rolled out an agent that could close tickets on its own. The demo was genuinely good. Twelve hand-picked tickets, clean terminal output, a round of applause. Six weeks later the agent was quietly demoted to drafting replies after it issued the same refund twice and marked a billing dispute as resolved without ever touching the account.
The engineering wasn’t the problem. The measurement was. Nobody had asked the boring question: how do we know this works, on which inputs, and how often?
AI agent evaluation is the discipline of answering that question with evidence instead of vibes. It’s also the part most teams skip, because building an eval suite produces no demo and wins no internal launch. It just stops you from shipping something that breaks quietly. If you’ve read about what separates a working demo from a system you trust, evaluation is the mechanism that does the separating.
Four things every agent eval has to capture
Traditional software testing checks one thing: did the output match the expected value? Agents need at least four measurements, and teams that track only the first one get burned by the other three.
- Task success. Did the agent reach the goal, judged against a written definition of done rather than a gut feel?
- Trajectory quality. Which tools did it call, in what order, and how many steps did it burn getting there?
- Cost and latency. Tokens spent, tool calls made, and p95 wall-clock time per task.
- Safety and side effects. Irreversible actions taken, data leaked, records written in the wrong place.
Multi-step tasks also deserve partial credit. An agent that books the outbound flight but fails on the return leg is not the same as one that books nothing at all, and a binary pass/fail score flattens a distinction you’ll need later when you’re triaging failures.
Build the eval set before you build the agent
This ordering matters more than any tooling choice. Once an agent exists, everyone unconsciously writes test cases the agent already passes. Start with 50 to 200 real tasks pulled from production logs, support exports, or sales transcripts, and include the ugly ones: typos, multi-intent requests, missing account numbers, a customer who changes their mind halfway through.
Split them. Roughly 70% as a dev set you iterate against, 30% held back as a test set you touch only before shipping. Without that holdout you end up with an agent that scores 94% on the exact cases its prompts were tuned on and 61% on anything new.
Where the cases actually come from
Mine the last 90 days of human-handled work. The tickets your best rep resolved in four minutes are your easy tier. The ones that got escalated twice are your hard tier. Then add a red-team pass: someone on the team spends two hours deliberately trying to make the agent do something stupid, and every successful attempt becomes a permanent test case.
Score the path, not just the destination
An agent that reaches the right answer through an absurd route is a time bomb. Classic example: an agent that looks up orders by calling list_orders with no filter and paging through the results until it finds a match. On a test database of 10,000 orders that finishes in seconds and passes every check. On a customer with 200,000 orders it times out, and the eval says nothing because the final answer was correct.
Trace every run. Store the full sequence of tool calls, arguments, and intermediate reasoning. Then write assertions against the sequence, not only the outcome: no unfiltered list calls, no more than six steps for tier-one tasks, no write operations without a preceding lookup. The agent loop nobody warns you about is exactly what trajectory scoring surfaces.
One run tells you almost nothing
Agents are stochastic. Run each task at least three times at production temperature, five if you can afford it, and report two numbers: pass@1 (how often a single attempt succeeds) and a consistency rate (how often all five attempts succeed).
The gap is where the pain lives. An agent at 90% pass@1 looks great until you notice its five-for-five rate is 59%. That means roughly four in ten tasks will fail at least once during a five-step workflow, which is what a customer experiences. Successful teams quote the consistency number in reviews, not the flattering one.
Cost and latency belong in the score
Accuracy gains are not free. Put dollars and seconds on the same dashboard as pass rates, because a three-point accuracy improvement that doubles token spend and triples latency is usually a losing trade.
Concrete version: an agent burning 40,000 input tokens per task at $3 per million costs about $0.12 a run. At 200,000 runs a month that’s $24,000, and a prompt change that adds 15,000 tokens of context silently adds $9,000. Track tokens per task, tool calls per task, and p95 latency, and treat all three as regression targets alongside correctness.
LLM judges are useful and easy to fool
For open-ended outputs, a model grader is often the only practical option. It’s also biased in predictable ways: it prefers longer answers, favors the option shown first in a pairwise comparison, and slightly prefers text it wrote itself. Shuffle the order, use pairwise comparisons over absolute 1-10 scores, and write a rubric with explicit criteria rather than asking “is this good?”
Then calibrate. Hand-label 100 examples yourself and measure how often the judge agrees with you. If agreement sits below roughly 80%, the judge isn’t ready to gate a release. And never use a judge for something a deterministic check can verify. Whether a refund amount equals $47.50 is a unit test, not a judgment call. Getting the grading prompt right is its own craft, closely related to the prompt engineering that separates a reliable agent from a confident liar.
Offline evals catch regressions. Online evals catch reality.
An offline suite runs on fixed tasks and tells you whether your last change broke something. It cannot tell you that real users phrase requests in a way you never imagined. For that you need production tracing, shadow mode, and a canary.
Shadow mode runs the agent alongside the human workflow without letting it act, and compares outputs. Canary releases it to 5% of traffic with a kill switch wired up. Then every production trace gets logged, sampled, and reviewed. Keep a separate bucket for tasks that failed because the environment was flaky rather than the agent being wrong, otherwise your scores drift for reasons nobody can explain.
Evaluate the vendor, not just the model
If you’re buying rather than building, the same logic applies to the sales process. Ask for their eval harness, their pass rates, and permission to run your own 50 tasks against a sandbox before signing. A vendor who can’t produce a scored eval on your data is selling a demo, which is the central test in how to pick an agent builder that ships real work. Read third-party write-ups the same way, since glowing reviews usually describe the onboarding call rather than month four, a trap covered in how to read agent software reviews without getting fooled.
What a working eval loop looks like
The teams that do this well aren’t running anything exotic. They’re just consistent:
- Every production failure becomes a test case within 48 hours, with the trace attached.
- The eval suite runs in CI on every prompt, model, or tool change, and blocks merges when scores drop more than two points.
- A scoreboard sits in the team channel showing pass@1, consistency, cost per task, and p95 latency.
- Someone reviews 20 real traces by hand each week, because aggregate numbers hide the failure modes that matter.
One detail that separates mature teams from the rest: they keep a changelog of every eval case added and why. Six months in, that log is the institutional memory of everything the agent has ever gotten wrong, and it’s worth more than any model upgrade. The scoreboard doesn’t need to be pretty. It needs to be the first thing anyone looks at before saying the word “ship.”