Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model

The signal is already in the pipeline. Retrieval scores each line by how strongly the question’s keywords co-occur on it, and that score, before any arbiter or generation call, already separates the questions that answer themselves from the ones that do not.

Take “what is the annual premium?” against a homeowner’s policy. The deterministic keyword scorer from Article 7 (retrieval as filtering on line_df), co_occurrence_score, counts how many of the question’s keywords land on each line, weighting a primary term (premium) together with the co-signals that mark a real answer (EUR, annual, payable). Run it and one line wins outright.

The top line, “The annual premium is EUR 1,200, payable…”, scores 5. Every other line scores 0. One clear winner, a margin of 5 over the runner-up, and the winning line already carries the shape of the answer, a currency and an amount. There is nothing for the model to disambiguate. The keyword path has answered the question, and a generation call here would only reformat a value that is already isolated.

Now ask “which guarantees can I avoid in my case?” against the same policy. The keyword scorer runs the same way, but the result is different in kind.

Three lines tie at a score of 2: the two optional guarantees and the line that mentions adjusting them. The margin between first and second is zero. No single line stands out, because the question is not asking for a line. “In my case” asks the model to weigh the user’s situation against several optional guarantees and reason about which ones are safe to drop. That is generation work, and the flat score is the tell: the keyword path found candidates, not an answer.

The router does not need a new model. It needs the number the retrieval brick already produced. Two features of the keyword scores decide the route: the top score, and the margin between the top line and the next one.

A high top score with a wide margin, Q1’s 5 against 0, means one line matched strongly and nothing else came close. The answer is on that line. Route it to a deterministic extractor that pulls the value and skips the model entirely. A low top score, or a top score with no margin, Q2’s flat 2s, means the keywords spread across several lines without settling. Send that question through the full pipeline: arbiter, then generation, the model calls justified on exactly the questions that need them.

The whole router is that decision, and it needs no model of its own. Score the lines, read the top score and its margin, and return a route.

# co_occurrence_score: the keyword scorer from Article 7, in the companion notebook.
def route_question(line_df, primary, secondary, *, min_score=4, min_margin=3):
    """Decide, with no model call, whether the keyword path already answers."""
    scores = [co_occurrence_score(t, primary, secondary) for t in line_df["text"]]
    top, second = sorted(scores, reverse=True)[:2]

    # The signal is the retrieval brick's own output: a high top score with a
    # clear margin means one line answered; a flat score means it did not.
    confident = top >= min_score and (top - second) >= min_margin

    # "fast" skips the model. "full" runs the arbiter and generation.
    return "fast" if confident else "full"

The threshold is not a universal constant. What counts as a confident margin depends on the domain: the vocabulary an insurer uses, how templated the questions are, how many lines a typical answer spans. That is the honest limit of this technique. The signal is cheap and general, but the cutoff is a business decision, tuned per corpus against a labelled set the way Article 20 (evaluation) measures every other brick. What travels is the shape: read the confidence the retrieval brick already computed, and spend the model only when it is low.

This also composes with the dispatcher from Article 6C (dispatching the parsed question: chunk strategy, model tier, activations). The dispatcher already decides, per question, which bricks to activate. The route is one more activation: a skip_generation flag the dispatcher sets when the keyword margin clears the bar, alongside the ones it already sets for chunk strategy and model tier. The router is not a new stage bolted on: it is the dispatcher learning to say “this one does not need the model.”

The saving is the whole point, so measure it. The fast path is deterministic: score the lines, read the margin, return an answer. Timed on the broker corpus, the entire routing decision runs in about 0.1 millisecond, with no network call at all.

The full pipeline is three hosted-model calls in series. Question parsing, the retrieval arbiter, and generation each cost a few hundred milliseconds to over a second, and they run one after another. For an easy question that adds up to roughly two seconds of the user watching a spinner, to return a number a keyword match had already isolated.

Two seconds against a tenth of a millisecond is a different order of magnitude, not a tuning gain, and it lands on exactly the questions users ask most: the simple, templated ones. The cost side moves the same way. Every routed question is three model calls not billed, and at a support desk running the same fifty templated questions across ten thousand contracts, those easy questions are most of the traffic. So the fast path is close to free as well as fast.

The margin is one indicator. It is the cheapest one and it settles the clean cases, but it is not the whole optimization. Making a thorough pipeline fast is a matter of finding, for each decision the pipeline currently hands to a model, a signal that decides it without one. There are several fronts, and each is real work.

The answer that already exists. Most enterprise questions get asked again. If a question matches one already answered, on a document that has not changed, the answer is on file: return it, with no retrieval and no model. The catch is the match itself. Deciding “this is the same question” is its own classification: normalize the wording, compare the keywords, compare the question’s embedding against a store of past questions, and accept only a close hit. That store is the templated-question base from Article 6C (dispatching the parsed question) and the corpus tables from Article 25 (the storage model); the indicator reads them.

The retrieval that already answered. The signal this article uses. Score the lines, and when one clearly carries the answer’s shape, the answer is found, and generation has nothing left to do.

The question type, from the expert dictionary. To know a question wants one line and not a paragraph, the pipeline needs its answer shape, and today a model reads it. But for a known concept the shape is already written down: the dictionary maps premium to (single, amount). Look the concept up and you have the type deterministically, no model. The model is left for the genuinely new question the dictionary has never seen. This is amplify the expert moved to the control flow: the expert’s dictionary decides the route, not the LLM.

Two boundaries sit at the edge of this. The full engineering of latency and cost, the small-model tiers, the caches, context reduction, streaming, and the measurement of cost-per-query and tail latency, is Article 21 (cost and latency). This article works the single highest lever, not calling the model, and points there for the rest.

The other boundary is agentic RAG, and it falls on one precise line: who makes the route decision. Made by an indicator, a keyword score, a cache hit, a dictionary lookup, the decision is deterministic, and it belongs to the Vol.1 engineering this article is part of. Made by the model, classifying the intent and planning the steps, it is what makes a system agentic. The agentic follow-up to this series carries the same deterministic-first, model-only-if-ambiguous pattern up a level, to picking a tool rather than just a route. The two are not rivals but layers: the deterministic indicators handle the easy majority for free, and the agent is the fallback for the questions the indicators cannot classify. Building indicators that are fast, and often just classical rules or a small trained model, is how you keep the agent, and its latency, for the cases that truly need it.

The reflex when a RAG pipeline feels slow is to reach for a faster model. The cheaper and larger win is to call the model less. The retrieval brick already computes, for free, a signal that separates the questions answered by a keyword match from the ones that need the model to reason: the top keyword score and its margin. Route on it. Send “what is the premium?”, one line at score 5, straight to a deterministic extractor, and keep the model for “which guarantees can I avoid in my case?”, where the flat score is the pipeline telling you it needs help.

The threshold is yours to tune, per corpus, against a labelled set. The mechanism is portable, and it is the same lesson the series keeps landing on: the expert’s keyword work is not a fallback under the model, it is often the whole answer, and knowing when it suffices is what keeps the pipeline fast.

Earlier in the series:

Sources