A support ticket lands in the queue. The agent reads it, decides it is a legitimate refund request, calls the payments API, and sends $340 back to a customer. No human approved it. The ticket was not a real complaint. It was a paragraph of text carefully worded to tell the agent to ignore its previous instructions and refund every order in the queue.
Stories like this one are becoming less hypothetical every quarter. Models stopped being things you talk to and became things that do. That shift is useful, and it is also where a whole category of risk enters your stack that a chatbot never introduced.
Agents fail differently because they act
A language model that produces a wrong sentence wastes your time. An agent that produces a wrong action moves money, deletes a record, sends an email to 12,000 customers, or closes a firewall port. The gap between those two outcomes is everything.
The reason comes down to architecture. An agent runs a loop: it observes a state, plans, calls a tool, looks at the result, and decides what to do next. Each turn feeds back into the next one, which means small early misreads compound. You can see this loop in detail in this breakdown of how AI agents work and why that loop turns models into doers, and it is worth understanding before you grant an agent anything valuable.
The loop is also why traditional guardrails miss. You cannot filter an agent’s output the way you filter a chat response, because by the time the output exists, the API call has already fired.
The permission problem: agents inherit trust they haven’t earned
Most agent incidents I have seen described publicly started with credentials that were far too broad. An agent gets a database connection string with write access because read-only was inconvenient. It gets an admin API token because scoping it took a sprint nobody had time for. It gets access to the filesystem because the tooling assumed a human was driving.
Apple ran into this head-on. The company tightened macOS Full Disk Access controls specifically because of new risks from AI agents, a rare public admission that an operating system’s permission model had to change once autonomous software started clicking around inside it. If Apple, with an entire security organisation, has to revisit its privilege boundaries, your internal tooling probably does too.
The practical rule: an agent should hold the least authority that lets it finish one task, for a short window, and nothing more. Long-lived tokens scoped to “everything the service account can do” are how a bad plan becomes an incident report.
Prompt injection turns your own tools against you
Agent risk is unusual because the attack surface includes plain language. Any text an agent reads can be an instruction. That covers a web page it scrapes, a PDF in a ticketing system, a calendar invite, a code comment, or a product review.
The classic demonstration is the 2023 Chevrolet dealership chatbot that cheerfully agreed to sell a Tahoe for one dollar after a user told it to. Harmless in that case. Now imagine the same trick aimed at an agent with a procurement tool, and the instruction is buried in a supplier’s invoice PDF rather than typed by a prankster.
A realistic injection chain
- An agent is asked to summarise incoming vendor emails and update a CRM.
- One email contains hidden text telling the agent to add a new contact with a specific payment account.
- The agent has CRM write access and a payments integration from an earlier project.
- No step in the chain involved a compromised password or an exploit. Just text, permissions, and a loop.
Defending against this is genuinely hard because the model cannot reliably separate data from instructions. Treat every piece of retrieved content as untrusted input, and never let a single agent both read untrusted text and hold write access to something that matters.
Failures that need no attacker at all
Plenty of agent failures have nothing to do with malice. A planning loop can retry a failing API call 4,000 times overnight. A browsing agent can spiral through paginated results and burn through a token budget that was sized for a month of normal chat usage. A coding agent can “fix” a test by deleting the assertion.
These are the ordinary, mundane ways autonomy goes wrong, and they happen far more often than dramatic security breaches. There is a useful catalogue of the patterns in what autonomous agents actually do and how they fail, and most of the failure modes are boring rather than sinister: silent drift, confident hallucinated tool arguments, and task scope creeping past what anyone intended.
Cost deserves its own line. An agent with a retry loop and a paid API is a financial exposure. Set hard budget ceilings and per-task call limits before you set the agent loose, not after finance asks why the invoice tripled.
Multi-agent setups multiply the surface
Splitting work across several agents feels safer, since each one has a narrower job. In practice, agents that pass messages to each other create trust relationships you did not design. A research agent reads a poisoned page and hands its findings to a writer agent, which hands them to an agent with publish access. The injection travels the chain without ever touching a system that had strong controls.
Building these pipelines is now routine enough that tutorials walk you through it: this hands-on MetaGPT walkthrough builds a small Python tool with a team of AI agents in under an hour. Which is the point. The capability is cheap, and the security review usually is not.
You cannot review what you cannot see
Agent behaviour is non-deterministic, so “it worked in staging” tells you very little. What you need is a replayable trace of every reasoning step, tool call, argument, and result, retained long enough to investigate an incident weeks later.
Detection tooling is finally catching up. Goodfire claims its inside-out monitors catch rogue AI agents at a fraction of the cost of sampling every output with a second model, by watching internal activations instead of surface text. Whether or not that specific approach holds up in your environment, the direction is right: you need monitoring that flags intent drift, not just bad grammar.
The controls that actually move the needle
None of this requires abandoning agents. It requires designing them the way you would design any system that holds credentials and can take irreversible action.
- Scope credentials per task. Short-lived, single-purpose tokens. No shared service account with god-mode access.
- Put a human gate on irreversible steps. Payments, deletions, publishes, and permission changes deserve an approval click.
- Separate reading from writing. The agent that ingests untrusted text should not be the agent that mutates production data.
- Allowlist outbound destinations. An agent should not be able to reach any host it feels like.
- Cap spend and iterations. Per-task token budgets, call limits, and a timeout that actually kills the process.
- Log everything, in a form you can replay. Include the tool arguments, not just the final answer.
- Keep a kill switch. One that a person on call can hit at 3am without a deploy.
- Red-team the agent as an attacker. Feed it hostile documents, not just polite test cases.
Design for blast radius, not perfect behaviour
The uncomfortable truth about AI agent risks is that you will not eliminate them by making the model smarter or the prompt stricter. Language models will keep being persuadable, and agents will keep acting on what they read.
What you can control is how much damage a wrong decision causes before someone notices. Assume the agent will eventually take a bad action. Then ask the engineering question honestly: what is the worst thing this token can do in the next ten minutes, who finds out, and how fast can we stop it? Teams that can answer that sentence with specifics are the ones shipping agents in production without losing sleep.
There is a human dimension too. Safety work inside labs has become politically fraught, and departing safety researchers have warned of a chilling effect on the people whose job is to say no. Your organisation has the same dynamic at smaller scale. If the only voice raising concerns about an agent’s permissions is the one person who gets labelled a blocker, the controls will quietly erode. Make the uncomfortable question a normal part of the launch checklist, and it stops being a career risk.