Twelve agents, one shared inbox. That was the setup inside a support operation I spoke with last spring: an order-lookup agent, a refund agent, an escalation agent, a tone-checking agent and eight more, each one solid in isolation. A single ticket arrived and got touched by four of them inside ninety seconds. Two issued refunds. One escalated to a human that another agent had already closed out. Nothing was broken. Everything was.
The models were fine. The wiring between them wasn’t. That gap, between a good agent and a good system of agents, is exactly where AI agent orchestration lives. It is the least demo-friendly part of the stack and the part that decides whether you ship or stall.
What AI agent orchestration actually means
Orchestration is the layer that answers four questions on every step: which agent runs next, with what context, under what permissions, and what happens if it fails or contradicts the agent before it. Routing, shared state, error handling and a paper trail. Nobody puts it in a pitch deck.
An LLM alone is a function. Text in, text out. Give it tools and a loop and you have an agent. Chain five of those together and you have quietly built a distributed system, which brings distributed-system problems along for free: partial failure, duplicate work, race conditions, and no obvious place to look when something goes sideways.
One agent versus many
Most teams split into multiple agents about two months too early. A single agent with twelve well-described tools will outperform a five-agent committee for a long time, because there’s no handoff to lose information in. The reasons to split are narrow and specific: different permission scopes, genuinely different models, work that must run in parallel, or context windows you cannot fit. “It feels cleaner” is not on the list.
The coordination tax, and why it compounds
Every agent you add doesn’t just add capability, it adds edges. Three agents means three pairwise relationships to reason about. Eight agents means twenty-eight. That’s the coordination tax, and it shows up as a specific set of failures long before it shows up in your architecture diagram.
- Duplicate side effects. Two agents both act on the same record. Refunds, emails, ticket status changes. The expensive kind of bug, because it’s visible to customers.
- Context drift. Agent three never learns what agent one discovered, so it re-queries the CRM and gets a stale answer.
- Silent contradiction. A later agent overwrites an earlier decision with no record that a conflict ever happened.
- Handoff loops. A routes to B, B decides it isn’t its job, B routes back to A. Burns tokens until something times out.
- Runaway cost. Retries multiply quietly. A pipeline that costs four cents per run costs forty when a downstream call starts failing.
The last one bites hardest because it’s invisible. You don’t notice until the monthly invoice arrives and someone asks why token spend tripled in a week where traffic was flat.
The orchestration patterns worth knowing
Almost everything in production maps to one of three shapes, or a hybrid of them.
Supervisor (hierarchical)
A lead agent decomposes the request, delegates to specialists, and assembles the answer. One place to set policy, one place to log, one place to enforce a budget. This is the default for a reason: it gives you a single choke point for guardrails. The cost is that the supervisor becomes a bottleneck and a failure domain, so it needs a hard step limit and a fallback path.
Sequential pipeline
Fixed order: retrieve, draft, check, send. Boring, cheap, and dramatically easier to test than anything dynamic. If your workflow already has a defined process, resist the urge to make it agentic. A pipeline with typed inputs between stages will beat a swarm on reliability every time.
Peer handoff
Agents pass control to whichever specialist fits, with no central coordinator. Flexible and fast for open-ended tasks, and the hardest to debug, because the execution path only exists at runtime. Good for triage and exploration. Poor for anything with money or compliance attached.
Pattern choice tends to be constrained by the tooling you pick, and not all frameworks support all three well. If you’re still comparing options, this breakdown of what each AI agent framework actually does is worth twenty minutes before you commit to one.
What the stack underneath looks like
Strip away the vendor names and a working orchestration layer has six parts: a router that picks the next step, a state store that survives between steps, a tool layer with per-agent permissions, a policy layer that can block an action, tracing, and budget caps. The first three get all the attention. The last three are what let you sleep.
State is where builds fall over
If agents pass context by stuffing the whole conversation into the next prompt, you’re paying for the same tokens six times and losing fidelity anyway. Better: a durable task record with structured fields, and each agent reading only the slice it needs. It makes runs replayable, which matters more than it sounds.
Observability isn’t optional
If you cannot replay a run step by step, you cannot fix it. Log every agent invocation with the input, the tool calls, the arguments, the output, the latency and the token count. Teams that skip this end up debugging multi-agent systems by re-running them and squinting, which is not a strategy. Larger shops often end up on a managed platform partly for this reason, and something like IBM watsonx earns its place through governance and audit trails rather than raw model quality.
How much autonomy to hand over
The useful mental model is a dial, not a switch. Read-only actions, like searching, summarising and drafting, can run unattended almost immediately. Write actions need thresholds. A refund under twenty pounds might clear automatically; above that, a human approves. Anything that touches an external party, sends an email or changes a contract gets a gate regardless of size.
Getting that dial right is most of the difference between a system that saves hours and one that generates cleanup work. There’s a good honest look at what autonomous AI agents can genuinely handle without a human watching, and the short version is: more than most people expect for retrieval, less than most people hope for irreversible actions.
The economics of many agents
Orchestration multiplies token spend in ways that surprise finance teams. A supervisor that summarises context for each of four specialists can easily produce five times the tokens of a single strong model doing the whole job. Two levers fix most of it. Cache aggressively, and route the cheap steps to cheap models.
That second lever is why small language models keep turning up in orchestration layers. Classification, intent routing, field extraction, tone checks: none of that needs a frontier model, and putting a 3B-parameter model on the routing step can cut per-run cost by an order of magnitude. Reserve the big model for the step where judgment actually matters.
Where orchestration quietly breaks
Most failures I’ve seen weren’t crashes. They were degradations nobody measured. A summarisation step between agents that shaved off the customer’s original complaint. An agent that re-fetched data every loop because nobody told it what the previous one already had. A retry policy that turned one bad API day into a four-figure bill.
The pattern behind all of them is the same: the system looked fine on the demo cases and rotted on the long tail. That is a well-documented shape of failure in this space, and it’s covered well in this piece on where AI automation pays off and where it breaks. Build the evaluation set before the second agent, not after.
The version that survives contact with production
Start with two agents and a supervisor. Give the supervisor a hard step limit of six and a rule that it must return a partial answer rather than loop. Log every step with token counts and tool arguments. Assemble 200 real cases, including the ugly ones, and re-run them on every prompt change. Set a per-run cost ceiling and alert when you cross 80% of it.
Then, and only then, consider a third agent, and only if you can name the specific failure the current design has. Teams that run multi-agent systems well are rarely the ones with the most agents. They’re the ones who can tell you, for any given ticket, exactly which agent touched it, in what order, and why.