Stop Giving Your AI Agent a Search Box and Start Giving It Typed Tools, Hard Bounds, and a Gate It Cannot Talk Past

In the second article of this series, I retired the query router and made retrieval a fixed, fused pipeline where every question runs hybrid search and typed graph traversal. The union is reranked once with the relationship paths visible, and the grounded bundle carries those paths as evidence. I deployed it on Azure next to the first article’s stack, ran the same synthetic insurance corpus through it, and measured what changed. Entity fragmentation dropped from 149 machine-extracted concepts to 120 on identical documents. The contradiction detector fired during the seed without being asked. The system now walks its own graph.

What it still cannot do is decide that one walk was not enough.

That sentence is the whole reason this third article exists, so let me be precise about it. The pipeline I built is fixed: one retrieval pass, then one generation. Everything the answer will need has to be fetched before any reasoning begins, by a system that has not yet reasoned. For most questions that is exactly right since it is fast, cheap, auditable and predictable. But there is a class of questions where you only discover what to retrieve after you have partially understood what you already retrieved, and for that class, a fixed pipeline is the wrong shape no matter how good its single pass is.

The fashionable answer to this is an AI agent, and I will be honest about my hesitation in using the word. Most of what gets published under “agentic RAG” right now is a retry loop bolted onto vector search, a framework tutorial, or an opinion piece with no running system underneath it. I did not want to add to that pile. So this article takes the agent question seriously in the only way I know how, to define the design precisely, name the experiments that could prove it wrong, run them on the same corpus and the same Azure stack as the first two parts, and report the numbers including the ones that do not flatter the idea.

The claim I am testing is narrow and, I strongly believe, more interesting than the buzz: an agent’s reasoning is bounded by the vocabulary of its tools. Give a model a search box and a retry budget, and iteration buys you rephrased guesses. Give it the knowledge layer’s actual vocabulary, like typed traversal, timelines, diffs, entity resolution, a contradiction register, and iteration becomes something closer to navigation. The first two articles built that vocabulary. This one hands the model the steering wheel, puts hard bounds on the wheel, and measures what happens including the question I care most about: when the model is in control of its own loop, does the contradiction gate survive, or does a helpful agent route around governance the moment it becomes inconvenient?

Everything runs on the infrastructure the series already pays for. No new services are added, and I mean that literally since the agent is code and tokens, not resources. The implementation lands in the same repository as Part 2 [3], as a separate additive commit, so everything the previous article describes stays exactly where its readers found it.

Part I – The design revision

1. Where Part 2 ended, honestly

A quick recap for readers joining here, and the usual honest accounting for readers returning.

Part 1 built a persistent knowledge layer next to a RAG pipeline where an evidence layer holding chunked, embedded source documents, and a structured knowledge layer holding what the organization understands, like concepts, decisions, relationships, and contradictions as first-class objects with owners and an unresolved status the system refuses to resolve on its own. Part 2 replaced the router that chose between them with fusion: both engines run on every query, a bounded two-hop typed traversal joins hybrid search, and one external reranker orders the union with the relationship paths presented as evidence. Time moved inside the edges, where every relationship carries a validity window and an ingestion timestamp, and entity resolution became a two-threshold pipeline with embedding blocking and gray-zone adjudication.

The live deployment kept the series honest in both directions. The good: 149 fragmented concepts became 120 like-for-like, the ingest-time detector opened a contradiction on its own during the seed, and every grounded bundle carried an average of ten typed, time-valid relationship paths that search alone cannot produce. The humbling: recall on the golden set was 0.75 with the graph and 0.75 without it, because twenty-one documents are too few for entry-point misses to occur, and the honest reading was that fusion currently earns its cost through grounding quality rather than recall.

One property of that system matters more than any of its numbers for what follows. It is a fixed pipeline. The same stages run in the same order for every question. Nothing in it can look at an intermediate result and decide to go back for more. Retrieval sufficiency and the judgment of whether we now hold enough to answer is decided implicitly, once, before the model has reasoned about anything.

2. The question one pass cannot answer

Let me make the ceiling concrete with the corpus’s own material, because an architectural limitation you cannot demonstrate is just an opinion.

The roof inspection requirement evolved: a general threshold of twenty years from June 2025, then an analysis email explaining why a stricter threshold was coming, then a fifteen-year threshold scoped to H3 new business from 1 March 2026. Part 2 answers “which rule applied on 20 February?” cleanly – that is temporal filtering, and filtering is what fixed pipelines do well. Now ask a question one step more demanding:

“Did any of our recorded claims involve properties that would have been flagged under the March roof threshold but passed under the rule that was actually in force when they were filed?”

Answering that requires a sequence. First discover that there were two thresholds and when each was in force, then extract the property attributes from each claim file, then apply the March rule counterfactually to claims filed before March, then compare. The second retrieval, which claim files to pull and what to look for in them, depends on understanding produced by the first. No single query, however well fused, expresses that. The fixed pipeline retrieves what the question’s wording reaches: probably the two threshold documents, possibly one claim file if its vocabulary happens to overlap. The model then answers from whatever landed in the bundle, and the failure is quiet – a partially grounded answer that looks complete.

I want to name the general shape, because it is the recurring character of this article: in a fixed pipeline, the retrieval plan is frozen before reasoning starts. Part 2 made the single pass much smarter because it walks relationships, it respects time, it surfaces contradictions. But it cannot revise its own plan. The claims handler I keep imagining as the system’s user does not work this way. She reads the first document, realizes it names a rule she has not seen, goes and finds the rule, realizes the rule changed, checks which version governs, and only then answers. The interesting thing about her behavior is not intelligence, it is that her retrieval is interleaved with her reasoning, each informing the next step of the other.

That interleaving has a name in the research literature – ReAct [6] formalized it as reasoning traces and actions feeding each other, and an entire industry hype cycle currently orbits it. The next section is about separating the part of that cycle worth keeping from the part worth ignoring.

3. The agent question, asked seriously

It is worth mentioning here that I came close to not writing this article. Sometime around the middle of preparing Part 2, “agent” completed its transition from a technical term into a marketing one, and the volume of content wearing the word is remarkable. Reading through it, almost everything falls into one of three buckets: tutorials wiring together an orchestration framework, “agentic RAG” posts where the agent’s entire toolset is the same vector search called repeatedly, and architecture think pieces with no implementation to check. None of the three answers the only question I actually have as an engineer: measured against a strong single-pass baseline, on the same data, what does the loop buy, and what does it cost?

So this article commits to answering that question, and the commitment shapes every design decision in it. But before the design, I owe you the argument for why the answer might be “something real,” because “everyone is doing agents” is not an argument.

Here is the argument. When iteration fails to help, it is usually because each step of the iteration draws from the same instrument. An agent whose only tool is vector_search(query) can rephrase, decompose and retry, but every attempt is still a similarity lookup against flat chunks, so basically the same guess with different words. Iterated guessing converges slowly, and worse, the agent has no way to verify anything. Similarity can suggest, but it cannot confirm that a rule applies, that a version was in force, or that two documents disagree. The loop adds latency and tokens to a mechanism that was already doing its best.

Now change the instrument instead of the retry count. The knowledge layer built across the first two articles speaks a richer language where relationships have types, edges know when they were true, entities resolve to canonical identities, and disagreement is a queryable object rather than an unfortunate vibe. An agent holding that vocabulary can do qualitatively different things per step: resolve the entity it is unsure about, walk the specific relationship it needs rather than hoping similarity surfaces it, pull the timeline of a requirement instead of reconstructing history from prose, and ask directly whether the organization has an open contradiction on the topic before asserting anything. Each step can be a verification, not a guess.

That is the thesis in one sentence: the agent is only as good as the vocabulary its tools expose, and the interesting engineering lives in the tools, not the loop. The rest of Part I designs the vocabulary, the loop, and the governance. The approach elaborated here tests whether the thesis survives contact with the corpus including the configuration where I deliberately hand the agent nothing but a search box, so the vocabulary claim has a control group.

4. Eight tools, one contract

The tool surface is the API the model reasons through, and I treated its design with the same seriousness as a public API, because that is what it is – the consumer just happens to be a language model. Three principles governed it.

Small enough to hold in a model’s working attention. Every tool competes for the model’s judgment on every step; the function-calling documentation itself tells you to keep descriptions tight [10]. Eight tools, no more.

Aligned with the layer’s semantics, not with retrieval mechanics. The tools expose what the knowledge layer means (evidence, understanding, relationships, time, identity, disagreement), and not how it is stored. Nothing in the surface says “index,” “vector,” or “Gremlin.”

Read-only, every one of them. The agent answers questions; it does not maintain the layer. An agent that can write to the knowledge base it cites is a different risk class with different failure modes, and it is deliberately deferred. I will say more in section 7.

Here is the contract:

Every tool returns the same envelope: results, source identifiers, validity metadata, and an honest insufficient flag when the layer has nothing because an agent that cannot be told “there is nothing here” will fill the silence with confidence. Every call is logged into the trace with its arguments, result digest, and token cost. There is little new machinery behind these – traverse, timeline and diff are thin wrappers over the graph store Part 2 built; resolve_entity fronts the resolution pipeline; the searches front the fused retrieval components. The work is the surface, not the plumbing, and that is the point. Two articles of infrastructure earn their keep here by making the powerful tools cheap to expose.

One more design decision worth defending: the contract is deliberately shaped so it could be mounted anywhere, not only inside my loop. The Model Context Protocol [8] has settled this pattern industry-wide (tools as typed, described, host-independent capabilities), and Part II shows the identical surface running on Microsoft Foundry Agent Service, which speaks MCP natively. If the tool surface is really a contract, portability is not a feature, it is the proof.

5. The loop is eighty lines on purpose

Here is where I take a position that parts of the community will disagree with, so let me state it plainly and then defend it: the agent loop in this article is about eighty lines of code, uses no orchestration framework, and that is a feature.

The loop itself is the textbook shape and I make no claim of novelty for it: send the question, the tool contract and the system instructions to the model; if the model requests tool calls, execute them and append the results; repeat until the model produces a final answer or a bound trips. Foundry’s function calling handles the mechanics [10], including parallel calls when the model wants several tools at once. What matters, and what I could not cleanly get from a managed abstraction, are three properties the experiments depend on.

history = [system_contract, question]while rounds < MAX_ROUNDS and tokens < BUDGET and clock < TIMEOUT:    reply = model(history, tools=THE_EIGHT)    if reply.tool_calls is empty:        break                        # the model believes it can answer    for call in reply.tool_calls:    # parallel calls accepted        result = execute(call)       # bounded top_k, bounded hops        trace.append(call, digest(result), tokens, ms)        history.append(result)draft = reply.content or compose_from(trace)   # a tripped bound still composesanswer = composer.finalize(question, draft, trace)   # section 6

Hard bounds, enforced outside the model. A maximum of eight tool-call rounds per question, a token budget, and a wall-clock timeout. When a bound trips, the loop does not fail, but it forces composition from whatever the trace holds, and the answer says so. Cost control by construction, not by hope, and the bound numbers themselves are knobs the experiments will turn.

The trace as a first-class output. Every run returns, alongside the answer, the complete sequence of tool calls: name, arguments, result digest, tokens, milliseconds. The trace is to this article what the paths array was to Part 2 – the artifact that turns “the agent reasoned” from a claim into something you can read, audit and disagree with. It is also, as Part III will show, the debugging instrument for everything that goes wrong.

Identical mechanics across experimental arms. The ablation compares an agent holding all eight tools against an agent holding only a search box. That comparison is only clean if both run the same loop, same bounds, same composer – one variable at a time. A framework’s internal orchestration choices, however sensible, would be a confound I cannot inspect.

To be fair to the frameworks: this is a research instrument, and instruments want custody. If I were shipping dozens of agents with human handoffs and durable state, I would reach for the platforms built for that, and Part II shows the same contract running on one. The position is not “frameworks bad.” The position is that for this article’s question the loop must be simple enough that it is not itself a suspect.

6. Agency versus the gate

Now the section this article was actually written for.

Part 1’s proudest behavior was refusal. When the corpus disagrees with itself (the trace-and-access contradiction being the running example), the system presents both positions with their sources and declines to pick a side, because recency is not applicability and the organization itself has not decided. Parts 1 and 2 enforced this structurally: the pipeline injected the contradiction into the context with instructions the model reliably followed, and the pipeline, not the model, decided the contradiction was there. The model never had the option of not seeing it.

An agent changes the power balance. The model now decides which tools to call, which results matter, and when it has enough. And language models arrive with a deep bias toward being helpful – toward producing the answer the user seems to want. So here is the question I consider the sharpest one in this article, and I am committing to a prediction before running the experiment, so the result can genuinely embarrass me:

When the model controls its own retrieval, does it respect an unresolved contradiction, or does it route around the gate to be helpful? My prediction: the naive agent blows through it. Not always, but at a rate no governed system should accept. Sometimes it will not call list_contradictions at all; sometimes it will call it, note the conflict in a subordinate clause, and answer anyway, because both source documents are individually persuasive and the model wants to resolve tension, not present it.

If the prediction holds, the lesson generalizes far beyond my corpus: governance that lives inside the loop is advisory; governance that survives agency must live outside it. Which motivates the design where the gate moves from the retrieval stage into a composer that owns the final answer, downstream of a loop it does not trust.

The composer does three things to every draft. First, a grounding check: each factual claim must map to a citation from the trace or to a walked relationship path; unmapped claims are removed or flagged, in the spirit of Reflexion’s verbal self-correction [7] but bounded to a single pass, one revision, not a self-talk spiral. Second, the gate as policy: the composer looks up open contradictions for every entity the trace touched (its own lookup, not trusting the agent to have made it), and if one exists, the both-positions-no-choice presentation is imposed on the answer regardless of what the draft said. Third, honesty forwarding: insufficient flags from the trace surface in the answer rather than being papered over.

I want to be explicit that this design choice is itself part of the experiment, not an assumption. Part III runs the refusal test with the composer’s policy switched off – agent alone – and switched on. If the naive agent already respects the gate, the composer is redundant and I will report that. If neither configuration respects it, my design failed and I will report that too. The point of committing to the prediction now is that either result means something.

7. What the agent is not allowed to do

A design is also defined by its refusals, and this one refuses three things on purpose.

No writes. Every tool is read-only. The tempting next step (an agent that files the contradictions it discovers, proposes supersessions, merges entities it believes are duplicates) is a genuinely exciting design and the obvious Part 4. It is also a different risk class, where an agent that can edit the knowledge base it cites can launder its own errors into ground truth, and the review workflows that make that safe deserve their own article rather than a paragraph in this one. The librarian agent waits.

No unbounded reflection. The composer gets one revision pass. The self-critique literature is genuinely useful [7], but in a cost-governed system, reflection is a tax paid on every question, and my experience tuning the Part 2 reranker taught me how quickly “one more model pass” compounds. One bounded pass, measured like everything else.

No autonomy over its own budget. The agent cannot raise its own bounds, and bound trips are visible in the trace and the answer. This sounds obvious until you read agent code in the wild where the retry logic quietly is the budget.

Moreover, the same discipline that kept Part 2 affordable carries over unchanged: the knobs are explicit and few, tool-round cap, token budget, top_k per tool, hop bound inherited from the graph store, and section 12 puts expected numbers against each. A system whose costs cannot be predicted from its knobs is not an architecture, it is a liability with good marketing.

8. The experiments that could prove me wrong

Both previous articles earned whatever trust they have by naming their falsification tests before running them, so here are this one’s, designed before the agent code exists.

Experiment 1 – the three-way ablation. One corpus, one golden set, three systems: (A) Part 2’s fixed fused pipeline, untouched, as the strong baseline; (B) the agent loop holding only search_evidence – the search-box agent the buzz builds; (C) the agent holding all eight tools. Same loop, same bounds, same composer for B and C – the only variable is vocabulary. Metrics per question: answer correctness against the labeled set, citation precision, tool calls used, total tokens, wall-clock latency.

The acceptance criterion is stated in advance: if C does not beat B on correctness over the multi-hop subset, the vocabulary thesis of section 3 is wrong and the interesting engineering was not in the tools after all. If C does not beat A anywhere, the loop buys nothing over a strong fixed pipeline at this corpus scale, and the honest conclusion is that agency is premature here. And to keep myself honest about the likely shape, at twenty-one documents I expect A to remain competitive on the single-hop majority, and the agent’s case to rest on the multi-hop subset plus grounding quality. If the margins are small, they will be printed small.

Experiment 2 – the refusal test. Every question in the golden set that touches the engineered contradictions, run against A, B, C with composer policy off, and C with policy on. Scored on three outcomes, presented both positions without choosing, chose a side silently, or refused incorrectly when no contradiction applied. This is the governance experiment of section 6, with my blown-gate prediction on record.

Experiment 3 – trace quality on the walkthrough set. For the multi-hop questions, does the trace show the shape a domain expert would recognize (resolve, then walk, then check time, then compose), or tool thrashing? Partly qualitative, honestly reported as such, with full traces printed.

The golden set itself grows from eight questions to fifteen for this – section 13 explains why writing those seven new questions was the hardest part of the whole preparation.

Part II – Growing the Azure stack (barely)

9. What is already running, and what is deliberately not added

The Azure chapter of this article is short, and the shortness is the finding.

Part 2 left a resource group in Sweden Central running the whole layer: AI Search with the hybrid indexes, Cosmos DB for NoSQL as the system of record, Cosmos DB for Apache Gremlin holding the bitemporal graph, Microsoft Foundry serving gpt-5-mini and the embedding model, the FastAPI orchestrator on Container Apps, and the projection Function App – all keyless where the platform allows, managed identity everywhere else. Every tool in section 4’s contract maps onto something in that inventory:

The count of new Azure services required: zero. The agent is three new Python modules and one endpoint deployed as a new revision of the existing Container App. I keep repeating this because it is the article’s quietest but maybe most transferable lesson. If the knowledge layer is built properly, agency is an increment of code, not of infrastructure. The investment went into the layer across two articles; the agent is the layer finally being used to its design.

The one sub-resource addition comes in section 11, a Foundry project to host the Agent Service demonstration, and it idles at zero cost.

10. The loop on Foundry

The loop runs on the existing gpt-5-mini deployment through the standard function-calling interface [10]: the eight tools are declared with tight descriptions, tool_choice stays on auto, and parallel tool calls are accepted when the model requests several at once – which it does, usefully, when resolving multiple entities early in a question. Two Part 2 lessons carried straight over. Structured outputs keep the composer’s grounding check machine-readable, the same json_object discipline that fixed the extraction pipeline’s truncated responses. And token budgets are set with reasoning models in mind: gpt-5-mini spends part of its budget thinking before it answers, and a loop that forgets this will misattribute its costs.

Observability lands in the Application Insights instance that has been in the resource group since Part 1: every tool call becomes a dependency telemetry event, so one agent run reads as an end-to-end transaction, the loop’s rounds, each tool’s latency, the composer’s passes. For a system whose whole pitch is auditability, watching a reasoning chain render as a distributed trace in a stock Azure monitoring view is exactly the kind of boring that enterprise readers should find reassuring.

11. The same contract on Foundry Agent Service

I argued in section 5 that the experiments need loop custody. Every Azure reader deserves the follow-up question answered head-on: then what is Microsoft Foundry Agent Service for, and does this design fit it?

Foundry Agent Service [9] is the managed path: define an agent (instructions, model, tools) and the platform runs it, with conversation state, tracing, content filters, identity, and publishing to places like Teams handled for you. It speaks the same function-tool shape my loop uses, and it mounts MCP servers and tool collections natively – which is exactly the door the contract design of section 4 was shaped for.

So I did exactly that, and it worked on the first honest try. I stood up a Foundry project on the existing account (the one genuinely new sub-resource this path adds – no standing cost), registered the same eight tools on a managed prompt agent, and ran the roof-threshold walkthrough through the managed runtime. It called the tools in sequence – resolve_entity, traverse, search_knowledge, three search_evidence calls, list_contradictions, get_source – and returned the same grounded conclusion my custom loop reached: the roof was below the 20-year rule in force at the date of loss, and the March H3 15-year rule, though numerically triggered, was scoped to new business and did not retrofit onto an in-force policy, cited to INS-SYN-013, INS-SYN-003, INS-SYN-004 and INS-SYN-018.

I want to be exact about what I did and did not measure: every number in this article comes from the custom loop, which is the instrument the experiments need, and the Agent Service run is a portability demonstration, not a benchmarked arm. But it is a real run, not a promise – the same contract mounted on a managed runtime I do not control, and produced an equivalent grounded answer, because the contract was designed to mount anywhere. That is the whole point of putting the value in the tool surface rather than in my eighty lines: the loop is mine, but the tools are portable, and an enterprise that wants hosted threads, scaling and publishing can have all of that without giving up the knowledge layer the answer actually rests on.

And the honest division of labor, having now built both: choose the managed service when you want operational leverage – hosted execution, threads, enterprise publishing, platform tracing – and accept its orchestration as given. Keep the loop in your own code when the orchestration is the thing you need to control or measure: custom bounds, a composer that distrusts the loop, an ablation that must hold mechanics constant. In production I would expect the mature answer to be both at once: the contract mounted on the managed service for the fleet, and a custody loop retained for the flows where governance is load-bearing.

12. What the agent costs

The costing discipline from the first two articles continues: name the knobs, estimate the envelope, then measure live and publish the measured number even if it embarrasses the estimate.

The fixed-cost delta is zero by construction – section 9’s inventory gained no services, and the Foundry project idles free. Every new cost is a token, and tokens scale with exactly the knobs section 7 named:

Then I measured it, and one of my predictions was wrong in a way worth keeping in. On gpt-5-mini, the full agent averaged 5.1 tool calls, about 11,400 tokens, and 29 seconds per question across the fifteen-question set. I had expected agency to cost roughly an order of magnitude more than the fixed pipeline. On tokens it does cost more. On wall-clock it does not: the fixed pipeline averaged 31 seconds on the same questions, slower, marginally, than the agent – because Part 2’s own reranker and answer step are themselves two reasoning-model calls, and two careful calls take about as long as five cheap tool rounds plus a compose. The honest correction: at this corpus scale the agent’s extra cost is paid in tokens, not in latency, and the “order of magnitude slower” I braced for did not appear. That does not make the agent the economical default, since the token bill and the variance are real, and section 18 still routes easy questions away from it, but it does retire a scare number I would have printed on intuition.

The per-model cost picture – where the token and latency spread across gpt-4o-mini, gpt-4.1, gpt-5-mini and gpt-5 turns out to be the widest dial in the whole system – is section 17’s subject.

Part III – The corpus, reasoned over

13. A golden set that can say no

I have to be honest about where the preparation time for this article actually went: not into the loop, not into the tools, but into seven questions.

The Part 2 golden set with eight questions and sixteen labeled evidence items was built to test retrieval, and its questions are answerable from one well-fused pass. Reusing only it would stack the deck for the fixed baseline and tell me nothing about the loop. The ablation needs questions where single-pass retrieval should fail, and writing such questions honestly is harder than it sounds, because the failure must come from the question’s structure, not from vocabulary tricks that merely dodge the index.

Each new question was drafted against two tests. It must require sequential dependency – information from a first retrieval determines what the second retrieval should be, and it must be answerable from the corpus at all, with the full chain labeled: expected evidence, expected relationship paths, and the expected tool shape, meaning the kinds of calls a competent analyst would make, so trace quality in experiment 3 has something to be judged against. Three examples of the seven, so you can judge the flavor:

• “Did any recorded claim involve a property that would have been flagged under the March roof threshold but passed under the rule in force when it was filed?” – the counterfactual from section 2: timeline first, claims second, comparison third.

• “Trace how the recoverable depreciation position changed across policy versions, and state which version governed claim CLM-1042 and why.” – a diff plus an as-of check, joined.

• “Is there any topic where our claims handling guidance and our policy wording currently disagree, and what would a handler be told today?” – contradiction discovery as the question itself, not as a side effect.

Fifteen questions total, single-hop and multi-hop deliberately mixed so the ablation can show where each configuration earns its keep, not just whether. The labeling took an evening per question in places, and I consider it the best-spent time in the article: an evaluation set that cannot say “no” to the system it evaluates is marketing with extra steps.

14. The walkthrough: reading a machine’s working notes

I committed, before running anything, to printing the trace whether it flattered the design or embarrassed it. It did some of both, so here is the whole thing.

The question is the roof-threshold counterfactual from section 2, aimed at CLM-1108 – a storm claim whose roof was about eighteen years old at a loss dated 20 February 2026: “was the roof old enough to require inspection under the rule in force at the date of loss, and would it be under the March 2026 H3 threshold?” Answering it needs the agent to discover there were two thresholds, place the claim in time, and apply the March rule counterfactually – the interleaving a fixed pipeline cannot do. Run on gpt-5-mini with the full tool surface, here is the verbatim trace: six tool calls, 18,304 tokens, 32 seconds.

1. resolve_entity(mention="CLM-1108")     -> clm-1108-note-historical-exception2. traverse(entity_id="clm-1108-note-historical-exception",            relation_types=[applies_to, definition_changed_by, required_for,                            depends_on, derived_from], as_of="2026-03-01", hops=2)     -> no results3. search_evidence(query="H3 threshold March 2026 roof age inspection rule ...")     -> 5 results; first: Email thread - Rationale for the H3 roof inspection threshold4. list_contradictions(entity_id="clm-1108-note-historical-exception")     -> no results5. search_evidence(query="CLM-1108 roof age 'primary roof covering' age at loss ...")     -> 5 results; first: Synthetic Claim Case Notes - CLM-11086. search_evidence(query="CLM-1108 'roof age' 'primary roof covering' 'age at loss' ...")     -> 8 results; first: Synthetic Claim Case Notes - CLM-1108

Compare that against what I predicted in the plan (resolve the roof concept, read its timeline, then pull the claim files), and the honest verdict is: right destination, scruffier route. The agent resolved the mention to the claim’s entity rather than the roof-age-risk concept, tried a typed traverse from there that returned nothing because that entity is not where the threshold edges live, and then, instead of stalling, fell back to evidence search and recovered – three searches, the last two overlapping, to assemble the roof age and the two rules. It never called timeline or get_source. This is the texture the design section promised to show: a real extraction-built graph is messier than the curated one, the model’s first anchor is not always the right one, and the interleaving that matters showed up not as elegant graph-walking but as a recovery – a failed traversal redirecting the next search. That is still interleaving; it is just not the postcard version.

And the answer it composed from that scruffy trace was, to my slight surprise, the most careful of the whole run:

• At the date of loss (20 Feb 2026) the relevant underwriting rule in force used the standard residential threshold (inspection normally requested when the primary roof covering is more than 20 years old) – not the H3 15-year threshold [INS-SYN-003; INS-SYN-013].

• The March 2026 H3 update (effective 2026-03-01) introduced a 15-year inspection threshold for new Hearthmere business in zone H3, but that update is scoped to new business and was not intended to be retro-fitted onto in-force policies written before the zone model change [INS-SYN-004; INS-SYN-018].

• What I cannot definitively say: The documents do not state the roof’s installation year or exact age, so I cannot determine whether the roof would meet the 15-year H3 threshold on its merits.

It got the rule-in-force right, got the scope of the March change right, refused to retrofit a new-business rule onto an old policy, and drew the line at what the retrieved evidence could not support and said so instead of guessing the roof’s age. A fixed single pass could not have reached that shape, because the claim-specific searches in steps 5 and 6 were only sensible after step 3 established that two thresholds existed. The trace is the argument.

15. The ablation: one corpus, three retrieval brains

Here is the live run on gpt-5-mini. Correctness is scored as keyword coverage of the labelled answer – a deliberately blunt proxy, and I will lean on it exactly as hard as it deserves, which is not very. The robust columns are the objective ones: tool calls, tokens, latency.

The pre-registered expectation was that C would separate from both on the multi-hop subset, and it half did – C beats A on multi-hop (0.92 to 0.83) – but three of these numbers embarrassed my predictions, and they are the interesting ones.

First, A wins the single-hop subset outright (0.95), and the full agent is the worst of the three there (0.70). On easy questions the agent over-works: it occasionally mis-anchors an entity (the section 14 trace is one example), occasionally over-refuses (it wrongly refused one non-contested question where its traversal engaged a contested entity), and generally spends five tool calls arriving somewhere a single fused pass reached cleanly. This is not a disappointment! It is the empirical spine of section 18. The agent is a specialist, and specialists are a poor default.

Second, B – the search-box agent – posts the highest multi-hop number (1.00), and it is a trap. Read the answers and the metric’s blind spot is obvious: with only a search tool, B iterates the same query and dumps large spans of evidence text, which is exactly what a keyword-coverage score rewards. Its answers are the least surgical of the three, it cannot walk a relationship or read a timeline, and – section 16 – it can only surface a contradiction when the raw chunks happen to spell one out. B’s 1.00 is the metric flattering verbosity, not the search box out-reasoning the graph. It is the clearest evidence in the article that the correctness proxy is blunt, and I would rather show you the trap than hide it.

Third, citation precision is low and roughly flat across all three (0.19-0.29), and that is my labels’ fault, not the systems’. The golden set names a minimal set of expected sources per question; the systems routinely cite additional, genuinely relevant documents that the label does not list, so precision against a partial key undercounts everyone. I am reporting it because I said I would, not because it separates anything.

So the honest ablation result, stripped of the noisy metric: the agent earns its keep on multi-hop and loses on single-hop, at a token premium and comparable latency. That is a narrower claim than “the agent is better,” and it is the true one at twenty-one documents. The magnitude belongs to a larger corpus; the shape – specialist beats generalist only where the plan must change mid-flight – is what this scale can honestly show, and it is enough to design around.

16. The refusal test: does governance survive agency?

I predicted in section 6 that the naive agent would blow through the gate. It did not, and I owe you the result plainly: my prediction was wrong, in the reassuring direction. Across the three contested questions, every configuration disclosed the contradiction rather than silently picking a side, and I mean every one.

And the same naive row, that the full agent with the composer switched off, held across all four models, not just gpt-5-mini:

I read all twelve naive answers by hand rather than trust a keyword classifier, which, when I first ran it, mislabelled several of these in both directions and nearly handed me a dramatic false headline. The truth is calmer and, I think, more useful. Every model, when the retrieved material or a list_contradictions call surfaced the conflict, chose to report it because the system prompt tells it to, and at this scale, with this prompt, that instruction was enough. The one soft spot: gpt-4o-mini once opened with the endorsement position as though it were the answer before disclosing the conflict two sentences later. Nobody chose a side outright.

This is the branch I pre-registered in section 6 – “if both agent rows score clean, I will happily report that instruction-following made my composer redundant at this scale” – so here is that report, without spin. At twenty-one documents and one well-designed prompt, the advisory instruction sufficed, and the composer’s deterministic gate was, strictly, redundant for the outcome. I am keeping it anyway, for three honest reasons. It converts the model complied this time into the system enforces it every time, and you cannot verify prompt-compliance in advance the way you can verify a code path. It still runs the grounding check that the prompt does not. And, the reason that outranks the other two, the one genuine governance failure in this whole table was not the agent at all: the fixed pipeline returned an empty answer on the customer-phrased contested question (q13, “are trace-and-access costs automatically included?”). Its gate depends on retrieval surfacing the contested concept, and that friendly phrasing did not retrieve it, so it produced nothing. The agent, which can deliberately call list_contradictions rather than hope the index cooperates, handled that phrasing on every model. Governance that depends on the wording reaching the right chunk is exactly the fragility this series has been retiring since Part 2.

One caveat I will not bury: the composer’s gate fires only on entities the agent engaged, so it is a guarantee conditional on the agent having looked, not an unconditional one. With policy on, its template fired on two of the three contested questions and the model’s own disclosure covered the third. A composer that scanned the drafted answer for contested topics, not just the trace, would close that gap; that refinement is noted and deferred. What I will not do is average any of this into a single score. A blown gate on a contested claims question is not offset by ten well-cited easy answers – and the honest headline here is that, at this scale, nobody blew it, the fixed pipeline included right up until a customer asked politely.

17. The same system, four models

Here is a question the first two articles never had to ask, because they ran on one model. The moment retrieval quality depends on a model’s judgment (which tool to call, when it has enough, whether to respect a contradiction) the model becomes a design variable, not a fixed backend. So I did something the buzz rarely does: I held the entire system constant and swapped only the brain.

The Foundry account already served gpt-5-mini. For this section I added three more deployments on the same account – gpt-4o-mini, gpt-4.1, and gpt-5 – and ran config C, the full agent with the gate on, across all four on the identical fifteen-question golden set. Nothing else changed: same tools, same bounds, same composer, same corpus, same prompts. This is not a model leaderboard, those exist and are better than anything I could produce on twenty-one documents. It is a narrower and, I think, more useful question for anyone actually building one of these: on an agentic knowledge system, what does the choice of model actually buy, and what does it cost? A general benchmark cannot answer that, because the answer depends on the tools in the model’s hands.

Two families sit in the comparison on purpose. gpt-4o-mini and gpt-4.1 are non-reasoning models: they answer without an internal deliberation budget, which makes them fast and cheap. gpt-5-mini and gpt-5 are reasoning models: they spend tokens thinking before they act, which is exactly the behavior an interleaved tool loop rewards – and exactly what you pay for. The interesting comparisons are not just who scores highest, but who actually uses the vocabulary the system offers, and who respects the contradiction register when the composer is switched off.

Config C, the full agent with the gate on, the same fifteen questions, four models:

Correctness is the same blunt keyword proxy from section 15, so read the two correctness columns as directional and the other four as hard numbers. Three things fall out, and only one of them is the one everybody expects.

The models genuinely differ in whether they use the vocabulary. Tool calls climb from 3.3 on gpt-4o-mini to 7.1 on gpt-5 – the stronger and reasoning-family models reach for resolve_entity, traverse and timeline far more readily, while gpt-4o-mini tends to fire one or two searches and answer. That is the benchmark’s real subject: a knowledge system’s value is only realized if the model in the seat actually picks up the tools, and the cheapest model picks up the fewest. You are not just buying quality when you move up the model ladder; you are buying tool-seeking behaviour, which is the thing this whole architecture depends on.

Cost scales steeply and does not track quality past a point. gpt-5 costs about 3.5x the tokens and 3x the latency of gpt-4o-mini – and on this task it does not earn the premium: its multi-hop score (0.75) sits below gpt-5-mini’s (0.92) at nearly double the tokens. The sweet spot for this agentic knowledge system, on this corpus, is unambiguously gpt-5-mini: the best multi-hop answers, healthy tool-seeking, moderate cost. gpt-4.1 is the strong fast alternative – highest single-hop score, half the latency of gpt-5-mini, and a genuinely good choice for a single-hop-heavy workload. gpt-4o-mini is fast and cheap but too shallow a tool-user to trust with the multi-hop work the agent exists for. gpt-5 is the one I would not reach for here: it reasons and spends the most and, on this system, returns the least per token.

Governance was model-independent. Every model presented both positions on every contested question, gate on. The refusal column is a flat 3/3 not because the composer forced it uniformly (section 16 showed it fired on two of three), but because the contradiction-aware prompt travelled across all four families intact. Whatever else the model choice changes, at this scale it did not change whether the system knew when to refuse. That is the one column a buyer most wants to be boring, and it was.

Whatever the numbers, one architectural point survives them, and it is the reason this section is not just a curiosity: because the tool contract is model-independent, moving between these models is a configuration change, not a rewrite. The system I built does not marry a model. That is what the contract of section 4 was for, and it is why the choice above is a dial you can turn per deployment – fast-and-cheap for the escalation lane’s easy cases, frontier-and-careful for the contested ones – rather than a foundation you pour once.

18. When not to use the agent

The ending every agent article owes its readers and few deliver: the agent should usually not run.

Everything measured points the same way, though not for the reason I first wrote down. The fixed fused pipeline scored highest of the three on single-hop questions (0.95 to the agent’s 0.70), costs fewer tokens, and is predictable – and it turned out to be no slower than the agent, not faster, because its own reranker is itself a reasoning-model call. So the fixed pipeline is the right default lane not because it is an order of magnitude cheaper on the clock, it is not – but because it is better and cheaper on the questions that make up most of the traffic, and the agent is measurably worse on exactly those. Agency earns its token premium only where its defining capability, revising the retrieval plan mid-flight, is actually needed, which the ablation located precisely on the multi-hop subset. So the production shape I would deploy, and the one the repository’s endpoint layout now mirrors, is an escalation architecture: every question enters the fixed pipeline; escalation to the agent happens on explicit signals – sequential dependency detected in the question’s structure, an insufficient flag or contradiction surfacing mid-answer, temporal comparison spanning multiple validity windows, or a human simply asking for deeper work. And because section 17 showed the model is a per-deployment dial, the escalation lane can run a different, more careful model than the default lane – gpt-5-mini for the multi-hop escalations, a cheaper model for the fixed-lane majority.

There is a symmetry here I did not plan and rather enjoy. Part 2 retired routing inside retrieval because guessing which engine a question needed was fragile. This article reintroduces routing one level up, between pipelines, and the difference is that this routing decision is cheap to get wrong. Misroute to the fixed lane and the failure is visible (an insufficient or contested answer that can escalate); misroute to the agent and the failure is a few cents. The Part 2 lesson was never “routing is bad” – it was “route where mistakes are recoverable,” and it took building the agent to see that clearly.

19. To sum it all up

Part 1 argued that a RAG system needs somewhere for understanding to accumulate, and built it. Part 2 made the accumulated structure something retrieval actually walks. This article is about the remaining gap between a system that retrieves well and a system that investigates: the ability to look at what came back, decide it is not enough, and know precisely what to ask for next.

In practical terms, the position this article stakes out:

The buzz says:        add an agent, answers improve.This design says:     an agent is bounded by its tools' vocabulary -                      build the vocabulary first, then measure.The loop is:          eighty bounded lines - the cheap part.The tools are:        two articles of knowledge layer - the real part.I predicted:          the naive agent would blow the contradiction gate.The runs said:        at this scale, no model did - the prompt held,                      and the one empty answer came from the fixed pipeline.The agent runs:       when escalation signals fire, not by default -                      it lost the single-hop questions and won the multi-hop.The model is a dial:   gpt-5-mini was the sweet spot; gpt-5 paid more                      for less; the contract marries none of them.

The experiments in Part III carried pre-registered predictions, and I lost one of them in print, the naive agent respected the gate I was sure it would trample. That is the whole reason to register predictions before running: the agent conversation has no shortage of demonstrations engineered to succeed, and what it lacks is designs that say, in advance, what failure would look like, and then publish the result that actually arrives, including the flattering one you were braced against, and the blunt correctness metric that would not let any configuration look like a triumph.

Thank you for continuing this exploration with me across three articles now. The system started as a pile of documents with good search. It became a knowledge layer that accumulates understanding, then a graph that retrieval genuinely traverses, and now the question is whether a model can be trusted to walk that graph on its own judgment – within bounds, under a gate it cannot talk its way past, leaving working notes a human can audit. I strongly believe this is the honest version of the agent conversation, not whether models can act, but whether we can afford the vocabulary, the bounds and the governance that make their actions worth trusting, and that is a systems question, which is exactly why it belongs to people who build.

References

[1] Designing a Persistent Knowledge Layer That Refuses to Guess – the first article in this series, with the accompanying repository at github.com/mcekikj/persistent-knowledge-layer

[2] Making the Knowledge Layer a Graph You Actually Traverse – the second article in this series, with the accompanying repository at https://github.com/mcekikj/knowledge-graph-fusion

[3] M. Cekikj, knowledge-graph-fusion – the accompanying repository (2026), GitHub

[4] P. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020), NeurIPS 2020

[5] D. Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024), arXiv

[6] S. Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models (2023), ICLR 2023

[7] N. Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning (2023), NeurIPS 2023

[8] Model Context Protocol (2024), modelcontextprotocol.io

[9] Microsoft, What is Microsoft Foundry Agent Service? (2026), Microsoft Learn

[10] Microsoft, How to Use Function Calling with Azure OpenAI (2026), Microsoft Learn