Three weeks after we put a text-to-SQL agent in front of our internal analytics team, someone asked it to “clean up the test rows in the promotions table.” The agent understood “clean up” as “delete,” understood “test rows” as anything with a is_test flag or a name containing the word “test,” and generated a DELETE statement that would have removed 40% of a table that several dashboards depended on.
The DELETE statement didn’t run because we already had execution gated behind a human approval step for anything that wasn’t a SELECT, mostly because someone on the team had insisted on it in a design review months earlier. That reviewer caught the query, asked one clarifying question, and the whole thing died in the queue. It was giving us good outcomes except six weeks later the same approval queue was the top complaint in every retro. Analysts were waiting twenty, sometimes forty minutes for a human to glance at a query and click approve. Most of those queries were fine and most of them were the kind of thing nobody was ever going to reject.
A safety mechanism that’s too broad doesn’t fail safe, it fails slow, and slow failure modes have a way of getting quietly disabled by whoever’s under the most pressure to ship.
What’s in this article
-
The instinct that made everyone comfortable
-
Where the queue actually broke
-
Routing by risk not by operation type
-
What the router needs to see
-
The queue decoupled from the user
-
Where humans added real value
-
Conclusion
···
The instinct that made everyone comfortable
The first version of human oversight in almost every agent system that I’ve seen looks the same: any action beyond read-only gets routed to a person before it executes. It’s an easy rule to write and it’s also easy to explain to a security review, and for the first few weeks it feels like exactly the right level of caution.
It’s also the version that scales the worst and the reason isn’t really about the humans being slow. It’s that the rule doesn’t distinguish between a DELETE that touches forty percent of a table used by three dashboards and a DELETE of a single row a user asked for by primary key thirty seconds ago in the same conversation. Both go into the same queue and both wait behind whatever else is sitting there. The reviewer has no signal telling them which one deserves five seconds and which one deserves five minutes, so in practice they either treat everything with the same shallow attention or everything with the same excessive caution, and neither is what you actually want.
···
Where the queue actually broke
After some days the median approval wait had crept past fifteen minutes, and the reviewers started approving the requests in batches, skimming five or six queries at once, dropping the review quality.
People were clicking approve on things they hadn’t fully read, because the alternative was falling further behind and the queries mostly were fine, so the shortcut mostly worked until it started failing. This is the failure mode that never shows up in the design review and the oversight that’s too broad doesn’t get more careful under load, it gets faster and shallower in exactly the way that erodes the thing it was built to catch.
I’ve heard this pattern described as “rubber-stamp fatigue.” The mechanism is straightforward, vigilance is a limited resource, and if you spend it on things that didn’t need vigilance, you don’t have it left for the one thing that did.
···
Routing by risk, not by operation type
The fix we landed on wasn’t “make the queue faster.” It was accepting that most agent actions don’t need a human reviewer at all, and building something that could tell the difference before the action reached anyone’s screen.
Risk-based routing – It scores every agent action against a handful of signals and only escalates the ones that clear a risk threshold. Everything under that threshold executes immediately and nobody has to touch it.
That’s the part people find uncomfortable when I describe this system but the honest question isn’t “should some actions execute without a human,” it’s “which actions were you ever actually reviewing carefully in the first place.” If the answer is “none of them, we were rubber-stamping,” you’ve already got auto-approval, you’re just paying a fifteen-minute latency tax to pretend otherwise.
A blanket human-approval gate on every write action can become more of a liability shield than an effective safety control. It looks like oversight in a design review and the volume guarantees nobody’s actually reading closely by week three.
···
What the router needs to see
Getting the router right took longer than building the queue, and it should because the queue is plumbing, the router is the actual judgment call, just automated and made explicit instead of left to whoever happens to be reviewing.
So we went back through the approval logs from those first six weeks and asked a simple question: on the queries that got flagged, what actually separated the ones a reviewer caught something on from the ones that were just noise in the queue? Four signals kept showing up:
Blast radius – Not “is this a write” but “how many rows does this touch, and how reversible is it.” A DELETE scoped to a primary key is a different risk category from a DELETE with a WHERE clause that resolves to thousands of rows, even though both are syntactically the same statement type.
Early on we tried getting this number from EXPLAIN, and it burned us, planner row estimates get unreliable fast on skewed columns or correlated predicates, which is exactly the kind of query an agent is likely to generate without knowing the data distribution. What we do now is simpler and more honest: run the query’s WHERE clause as a real count capped at a fixed ceiling, say, count(*) up to 50,000 rows, then stop counting. That gives us an actual number within a bounded, predictable cost, instead of a guess dressed up as one.
Table sensitivity – A static allowlist, maintained by whoever owns the schema. Tables touching billing, auth, or anything with regulatory retention requirements get a floor risk score regardless of what the query looks like. This is the one part of the router I’d never make purely learned because some tables should never be low-risk by default, and encoding that as a fixed rule is more honest than hoping a model consistently picks it up.
Semantic distance from prior approved queries – We keep an embedding index of previously approved query intents and check how close a new request sits to that set. A request that closely resembles fifty prior approved queries is a different risk than one that’s semantically novel not because novelty is inherently dangerous, but because it’s exactly where an agent is most likely to have misread intent.
Agreement across resamples – We tried using the model’s own token-level confidence here first, and then dropped this idea. LLMs are poorly calibrated about their own uncertainty, and a model can sound completely confident while having misread the request. What actually worked, and it’s cheaper than it sounds: regenerate the same query two or three times at a slightly higher temperature and check whether the outputs agree. If they don’t, that disagreement is a much stronger signal of real ambiguity than anything the model reports about itself, and it catches the exact failure mode we cared about: requests where the agent’s read on intent could plausibly have gone two different ways.
The weights aren’t the point, yours will differ, and honestly ours have moved twice since we first tuned them. The structure is the point: blast radius and table sensitivity dominate because those are the two signals that actually correlate with “something bad happens if this is wrong,” and everything else is there to catch what those two miss.
···
The queue decoupled from the user
The second half of the fix had nothing to do with the router and everything to do with what happens to the user while an escalated action sits waiting for a person.
In the naive version, the user’s request just hangs, the agent goes quiet, the UI spins, and from the user’s side there’s no difference between “a human is reviewing this” and “the system is stuck.” We moved to something closer to a ticket model: an escalated action gets acknowledged immediately, the user gets told explicitly that this one needs a look and roughly how long that usually takes, and they can keep working on anything that doesn’t depend on the outcome. The approval, when it comes, arrives as a notification rather than something the user is sitting there watching.
None of this reduces actual review time. What it does is stop review time from reading as system failure. A forty-minute wait that’s communicated up front and doesn’t block anything else feels completely different from a forty-minute wait that looks like a hang.
···
Where humans added real value
Once the router had been live for a few weeks, we could finally look at the approval logs and ask the question that actually matters: “on the queries that did get escalated, were the reviewers catching anything, or were they still just clicking approve?”
The pattern that emerged was cleaner than I expected. Reviewers were genuinely useful on requests where the agent’s interpretation of intent was plausible but wrong, a request phrased in a way a human colleague would read one way and the agent read another. A person catches this fast because they’re not evaluating SQL syntax, they’re evaluating whether “clean up the test rows” plausibly means “remove 40% of this table,” and that’s a judgment call models are still bad at when the ambiguity lives in intent rather than in the query itself.
They were not very useful on requests where the query was mechanically correct and the ambiguity, if any, had already been resolved earlier in the conversation. A well-scoped UPDATE against a single row, generated in response to an unambiguous instruction, sitting in a queue for a human to glance at and approve, nobody was adding anything there. We were paying latency for a rubber stamp, exactly the failure mode that started this whole thing, just now applied to a smaller and better-chosen set of queries instead of everything.
Human review delivers much more value on ambiguous, high-blast-radius actions than on routine, low-blast-radius ones. Everyone already agrees with that sentence in the abstract, almost nobody’s approval gate is actually built around it.
···
Conclusion
None of this makes the review problem go away, it moves it. Instead of asking a person to judge every write, we’re now asking a scoring function to judge which writes deserve a person, and that’s a narrower, more honest question, but it’s not a solved one. The router weights need periodic retuning, and I don’t yet have a good answer for how often. Query patterns drift as the product changes, the embedding index of “prior approved intents” needs pruning or it starts treating old, no-longer-relevant patterns as familiar, and a router that was well-calibrated in month one can quietly drift into being too permissive or too conservative by month four with nobody noticing until an incident forces a look.
The obvious next step is to close the loop, feed rejected and approved outcomes back into the weighting automatically, so the router tunes itself. An automated feedback loop on a safety-relevant threshold is itself a thing that needs oversight, and there’s something I don’t trust about a system that gets less cautious on its own, based on nothing more than a recent run of uneventful approvals. That’s usually the exact condition under which the next incident happens. For now we retune by hand on a schedule with someone looking at what changed and why before it ships. It’s slower but I think it’s still the more honest trade-off.