A chatbot that gets an answer wrong costs you a support ticket. An agent that takes a wrong action costs you a refund, a deleted record, or an afternoon with legal. That gap is why AI agent architecture gets argued about so intensely and settled so rarely. The interesting engineering isn’t in the model call. It’s in everything wrapped around it.
Swap the model in a well-built agent for a different vendor and it usually keeps working, maybe a little worse or better. Swap the architecture underneath a brilliant model and the whole thing falls apart by step four. The distinction between an assistant and something that acts on your behalf is structural rather than conversational, which is exactly the point made in this breakdown of AI agents versus chatbots. Here’s how these systems are actually put together, where they break, and which decisions deserve your attention.
The four parts every agent shares
Strip away vendor branding and almost every production agent has the same skeleton.
- A reasoning core: the model, plus the prompt, constraints, and output format it operates under.
- A tool layer: the functions, APIs, and data the agent can actually touch.
- Memory and state: what it knows right now, what it learned earlier, what survives the session.
- A control loop: the code deciding what runs next, when to stop, and what happens on failure.
Most teams pour their first month into the reasoning core and regret it. Reliability lives in the control loop. Safety lives in the tool layer. Both are ordinary software engineering, which is either reassuring or disappointing depending on what you were promised at the kickoff meeting.
The loop is the engine, and it needs a budget
The default pattern is reason, act, observe. The model thinks about the goal, picks a tool, reads what came back, and repeats. It’s simple to implement and it’s where nearly all runaway behaviour originates.
Add two things on day one. A step budget, because an agent stuck retrying a failing API will cheerfully make forty calls before anyone notices. And a termination condition that isn’t “the model says it’s finished,” because models are relentlessly optimistic about their own progress. Ten steps with a hard stop is a reasonable ceiling for most business tasks. When the budget runs out, return partial results and a plain statement that the task wasn’t completed. Silent failure dressed up as success is the worst possible output.
The mechanics of a task-queue loop are easier to grasp by reading code than by reading diagrams. If you want the version with nothing hidden behind a framework, this walkthrough of building a BabyAGI-style agent from scratch is a useful afternoon.
Planning: decomposition decides quality
Ask an agent to “handle the vendor onboarding” and you’ll get mush. Ask it to check the W-9, verify the bank details against the invoice, and draft the welcome email, and it starts performing. Granularity of the task breakdown matters more than which model version you’re running.
Static plans versus replanning
A static plan generates the full step list up front and executes it. Predictable, cheap, easy to audit, and brittle the moment step three returns something unexpected. Dynamic replanning regenerates the remaining steps after each observation. More capable, roughly 2–3× the token spend, and much harder to reason about when something goes wrong at 2am.
Most production systems land somewhere in between: a static skeleton for the parts of the workflow that are genuinely fixed, with replanning allowed inside individual stages.
The failure modes you’ll actually hit
Overplanning is common. Agents split a two-step task into nine steps and lose the thread of what they were doing. Plan drift shows up next: the agent completes a sub-goal, forgets the original objective, and starts solving an adjacent problem nobody asked about. Then there’s the retry loop, where the same tool is called four times with identical arguments and identical errors. Detecting repeated identical calls and forcing a strategy change is a five-line guard that saves real money.
Memory is at least three separate problems
“Add memory” is the vaguest requirement in the entire field. Break it apart and the engineering gets tractable.
- Working memory is the current context window. Everything the agent can see right now.
- Episodic memory is the log of what happened in this session or this account’s history.
- Semantic memory is retrieved knowledge: documents, past tickets, product data, typically in a vector store.
Big context windows tempt people into skipping retrieval entirely. Feed the agent the whole knowledge base and let it figure things out. It works, until you’re paying for 150k input tokens on every one of twelve steps. Retrieval with 5–8 well-chosen chunks usually beats a giant context on both cost and accuracy, because a smaller, more relevant context gives the model less to get distracted by.
Long sessions need summarisation checkpoints too. Past roughly 30 turns, raw transcripts become mostly noise. Compress to a structured summary and keep the last few exchanges verbatim.
The tool layer sets your autonomy ceiling
How much freedom you can safely give an agent is determined almost entirely by the tools you hand it.
Keep tools narrow and single-purpose. get_customer_by_email and issue_refund are safer than one manage_billing function that takes an action string. Separate reads from writes, and gate the writes behind approval when the blast radius is large. Make write operations idempotent so a retry doesn’t double-charge anyone.
Then write your error messages for the model, not for a developer. “Rate limited, retry after 30s” lets the agent adapt. “HTTP 429” teaches it nothing and it will try again immediately.
Multi-agent topologies: powerful, pricey, often unnecessary
Splitting work across specialised agents is genuinely useful when roles require different tools, different permissions, or genuinely parallel workstreams. A supervisor agent routing to a researcher, a writer, and a fact-checker can outperform a single generalist, and frameworks exist to make the coordination manageable, as shown in this hands-on MetaGPT build with a team of AI agents.
The costs are real, though. Every handoff is a chance to lose context. Debugging a five-agent pipeline means reading five interleaved transcripts to find one bad decision. Start with a single agent and split only when you can name the specific thing the split fixes. The commercial world is moving this direction anyway: Vesta’s $30M raise to bring agent swarms to mortgage lenders is a bet that coordinated specialists beat one big generalist in heavily regulated workflows where each step has its own rules.
Guardrails and tracing are the shipping decision
If you can’t replay a run step by step, you don’t have a system, you have a rumour. Log every prompt, tool call, argument, and observation with a run ID. Set token and dollar budgets per task and per tenant. Sandbox anything that executes code or moves money.
Safety practice in this space is still catching up with deployment reality, and the disputes inside the labs themselves are worth watching, including these warnings from fired OpenAI safety researchers about a chilling effect on internal scrutiny. Build as if external oversight is minimal, because it usually is.
Do the arithmetic before the first prompt
Take a ten-step task with an 8,000-token context growing by roughly 1,500 tokens per step. That’s about 150k input tokens for one completed job. At $3 per million input tokens, you’re at roughly 45 cents per task. Run 20,000 tasks a day and you’re spending $9,000 daily on input alone, before any output tokens or retries.
That number drives architecture. Cache the static system prompt and tool schemas. Route simple classification steps to a small model and reserve the big one for planning. Truncate aggressively. These choices routinely cut spend by 60–70% without touching quality.
Build the smallest loop that works
The pattern that succeeds repeatedly is unglamorous. One agent, four or five sharp tools, a step budget of ten, retrieval over a curated index, structured logging, and a human approval gate on anything irreversible. Run it on fifty real tasks and measure how often it finishes, how much it costs, and how bad the failures are.
Only then add replanning, extra agents, or longer memory. The architecture you can explain to a colleague in two minutes on a whiteboard is usually the one that survives contact with production. The one that needs a diagram with fourteen arrows rarely does.