Optimizing LLM Inference Costs in Multi-Agent Systems with Adaptive Model Routing

An Adaptive Model Router in front of your LLM pipeline can cut inference costs by up to 90%, without updating any of your agent logic.

In multi-agent systems, a naive design would be to use the same, most powerful LLM for all agents. For instance, the Planner, Researcher, Action, Analyst, Reporter — all use GPT-5-Pro or similar, regardless of whether they are doing a simple web search or constructing a detailed policy risk analysis of dense contracts.

In more realistic systems, the Planner creates a step-wise plan, distributing tasks among the downstream agents. This works well if the type of queries your system handles is consistent and predictable, in which case, agents can be statically assigned small, balanced, or large models based on their role.

What if the above conditions are not true? If you need a versatile assistant that handles varied queries; such as for different departments of an organisation, in which case the data sources, tools, and input context are best known at runtime. Static assignment breaks down.

This article explores an architecture for an Adaptive Agent Model Router. Rather than relying on a massive, monolithic Global Planner upfront, this system pushes planning downstream to each agent. It generates Just-In-Time (JIT) sub-tasks tailored to each active agent’s specific capabilities exactly when they are needed. By inserting a lightweight classification layer in front of this execution, the router evaluates these JIT tasks on the fly, dynamically assigning the right-sized LLM to the task while giving you granular visibility over your inference spend.

A Cheap Model to Choose the Right-Sized Model

The core insight is that entire planning does not need to be upfront. For a multi-purpose assistant, where the type of queries is unpredictable, building every possible scenario in a single Global Planner agent, along with tools, guardrails, output formats requires a lengthy, complex prompt. This becomes difficult to maintain and is highly prone to LLM hallucination. Furthermore, a global upfront planner is fundamentally “context blind.” It has to guess what a downstream task will require before upstream agents have even gathered the data. It also forces you to use the most expensive, powerful model just to generate the plan. Furthermore, massive upfront planning relegates every downstream agent to a mere executor of tasks, without taking advantage of the reasoning that an agent is capable of.

The second core principle is that classification is much simpler than execution. Figuring out whether a task is high-complexity or low-complexity requires far less intelligence than actually performing it. This means you can use a very fast and cheap model for routing such as a Gemini-flash-lite or a gpt-nano for task classification and LLM assignment, while still maintaining the accuracy needed.

Therefore, we can distribute the planning phase to each individual agent in the pipeline — the Researcher, Analyst, Critic and Reporter in the case demonstrated here. In this architecture, each active agent in the pipeline dynamically invokes a Planner Agent for its specific stage. When it is the Researcher’s turn, it passes its specialised mandate (Data Collection) to the Planner Agent, which considers the available data sources and generates a concrete set of sub-tasks just for that stage. The Analyst or Critic sees the full output of the Researcher to decide the LLMs for their tasks, mitigating the context-blindness. Because this planning call is strictly scoped to a single agent’s narrow role, it is an inherently low-complexity activity, requiring only a cheap, fast-tier model.

Then, as the active agent executes those generated sub-tasks, the router intercepts each individual task and runs a structured classification prompt through the lightweight model. That classification returns three scores:

These three scores are summed into a single routing score from 0 to 6, which maps cleanly to a model tier:

– Score 0–2 → Fast tier (e.g., gpt-5-mini)

– Score 3–4 → Balanced tier (e.g., gpt-5)

– Score 5–6 → Powerful tier (e.g., gpt-5-pro)

The context size dimension is particularly important in multi-agent pipelines because context accumulates. A Researcher agent starts with a small prompt. An Analyst inherits the Researcher’s output. A Critic inherits both. By the time the Reporter runs, it may be processing 7,000 or 8,000 tokens of accumulated analysis and critique, which alone pushes the routing score toward the Powerful tier, regardless of how simple the individual sub-task looks in isolation.

Architecture

The architecture for the adaptive model router is the following:

There are four moving parts:

Multi-Agent Pipeline: In this article, there are four sequential agents: Researcher gathers information, Analyst interprets it, Critic challenges the reasoning, Reporter synthesises it into a deliverable.

Planner Agent: Before executing any tasks, the active agent dynamically invokes the Planner Agent. The Planner takes the agent’s specific mandate (e.g., Data Collection) and generates a set of sub-tasks. Because planning prompts are inherently low-complexity, this step automatically routes to the cheapest Fast tier model.

Adaptive Router: Each generated sub-task is intercepted by the router before execution. The router uses a fast classifier model to score the task across three dimensions: complexity, reasoning, and accumulated context size. This ensures the right-sized LLM is assigned for the task for optimal balance of quality and cost.

Tier Models: Three models mapped to score ranges. The models themselves are configurable whereby, labels and pricing are set via environment variables, so the router is provider-agnostic.

The backend is a FastAPI application exposing SSE streaming endpoints. The frontend is a React/Vite app that consumes the stream in real time, updating agent cards and cost totals step by step as routing decisions arrive.

In a production setting, the router already has the incoming payload from the upstream agents. The classification criteria is derived as follows:

from pydantic import BaseModel, Fieldimport tiktoken# 1. The LLM only grades the qualitative dimensionsclass StepClassification(BaseModel):    complexity: str = Field(description="'low', 'medium', or 'high'")    reasoning: str  = Field(description="'low', 'medium', or 'high'")    complexity_explanation: str = Field(description="One sentence why this rating")# 2. The Router calculates the exact input token count deterministicallydef calculate_exact_context_size(prompt: str, accumulated_context: list[str]) -> int:    """Use tiktoken to get the exact token count of the payload before routing."""    encoder = tiktoken.get_encoding("o200k_base")        full_payload = prompt + "n".join(accumulated_context)    token_count = len(encoder.encode(full_payload))        if token_count < 2_000: return "small"    if token_count < 6_000: return "medium"    return "large"

The router then combines the LLM’s qualitative scores with its own deterministic context calculation to produce the final model tier.

COMPLEXITY_SCORES  = {"low": 0, "medium": 1, "high": 2}REASONING_SCORES   = {"low": 0, "medium": 1, "high": 2}CONTEXT_THRESHOLDS = {"small": 0, "medium": 1, "large": 2}TIER_THRESHOLDS    = {"fast": (0, 2), "balanced": (3, 4), "powerful": (5, 6)}def route_task(self, classification: StepClassification, exact_context_size: str) -> str:    """Sum the dimensions into a single score and map it to a model tier."""        # Sum the 3 dimensions    score = (        COMPLEXITY_SCORES.get(classification.complexity.lower(), 0)        + REASONING_SCORES.get(classification.reasoning.lower(), 0)        + CONTEXT_THRESHOLDS.get(exact_context_size, 0)    )    # Map score to tier    for tier, (lo, hi) in TIER_THRESHOLDS.items():        if lo <= score <= hi:            return tier                return "powerful"  # Safe fallback for edge cases

Experiment Results

I ran a variety of queries through the pipeline and a few snapshots are the following:

Semantic Context Inference

Query: What are the main benefits of adopting cloud computing for enterprise businesses, and what are the key risks to consider?

For the Researcher which is the first agent in the pipeline, there is no accumulated context from previous agents; this forces the router to estimate the data load from scratch for every step. When the classifier evaluates the identify_security_risks step, it correctly infers a small context window. However, when it evaluates the final synthesize_source_data step, it understands that synthesis requires passing all previously retrieved documents into the prompt. So the context score is increased to Medium, pushing the task over the threshold from Fast to Balanced. The router is going beyond grading only the task difficulty, it understands the data architecture of the pipeline.

Same Agent, Different Tiers

Query: Compare the long-term economic impacts of renewable energy versus fossil fuels on emerging market economies, including jobs, infrastructure costs, and energy security.

Same agent, consecutive steps, still a 14× cost difference. The first step asks the Analyst to aggregate basic quantitative data. The second step asks the Analyst to evaluate the correlation between energy transition pathways and long-term macroeconomic stability in emerging markets. The classifier rates the latter as High on both complexity and reasoning, escalating it to the Powerful tier. The context is classified to be Medium for all tasks since the researcher’s output flows into the Analyst, contributing to the classification. In this instance, the router evaluates the semantic weight of the specific task, instead of classifying everything as a general “Analyst” task.

High-End Reasoning

Query: Design a comprehensive framework for assessing the reproducibility crisis in machine learning research. Propose concrete standardization protocols, benchmarking practices, and institutional incentives.

Sometimes a query is genuinely dense. When evaluating the root causes and institutional failures driving the ML reproducibility crisis, there are no simple checklist tasks. Every single step the Critic agent generates requires deep domain expertise, multi-perspective synthesis, and abstract institutional policy analysis. In addition, the Critic receives detailed research and analysis from the Researcher and Analyst agents. The router therefore, honestly grades every step as High complexity and High reasoning, routing the entire workload to the Powerful tier. The router does not compromise quality to save cost.

The Pipeline Cost Comparison Across All Queries

The table shows adaptive router savings vs executing all steps with the most powerful model. As can be expected, the savings are not a fixed percentage, they scale inversely with the actual difficulty of the query. A straightforward enterprise query keeps almost every step in the Fast and Balanced tiers, slashing costs by over 90%. Dense, multi-domain macroeconomic queries genuinely require powerful models, and the system reflects that honestly. That justifies the core principle of this architecture; you pay only what each task requires using an objective classification criteria.

Conclusion

It is not necessary to use frontier models for each query, agent or task. As multi-agent systems become the standard architecture for enterprise AI, treating every task with the same expensive reasoning is financially wasteful and technically not prudent.

The Adaptive Model Router demonstrates that you don’t have to choose between cost-efficiency and cutting-edge performance. It is not always optimal to have a heavy Planner/Orchestrator dictating steps at the front of a pipeline when there is least clarity on the context that each task will need to operate on. By having each agent plan its tasks and decoupling the planning of a task from its execution, we can make the shift from static, role-based model assignment to dynamic, task-based routing for agents.

The result is a pipeline that intelligently scales its own intelligence. It confidently drops down to fast, cheap models for standard data retrieval and formatting, slashing costs by up to 90%. When it encounters genuine, multi-domain complexity, it scales up to the most powerful models available, refusing to compromise on quality.

In a production environment, this means that the most powerful AI solves your hardest problems, not summarize web searches. By evaluating the actual semantic weight of each task at runtime, the Adaptive Model Router ensures that the inference cost is measurable against the outcome.

Connect with me and share your comments at www.linkedin.com/in/partha-sarkar-lets-talk-AI