Put the Agent Inside the Workflow

, one of the first design decisions we have to make is:

Workflow or agent?

The workflow paradigm follows a sequence we define in advance. This makes the application very easy to understand, and gives us clear control over how information moves from one stage to the next. It works well when we know which operation should happen at each stage. For more open-ended questions, however, the next useful action may depend on what the system discovers along the way.

The agent paradigm, on the other hand, starts with a goal and decides which actions or tools to use along the way. This makes it more flexible when the solution path is uncertain. However, that flexibility also means we give up some control over how the solution unfolds.

But why make this choice for the entire application?

From what I see, in practice, many applications contain both kinds of work. Some stages might perform a known transformation, then the workflow paradigm is a good choice. Other stages might need to adapt based on intermediate results, which naturally calls for the agentic approach.

For these applications, a hybrid workflow-agent pattern is a better fit.

In this post, we’ll explore this hybrid pattern through a concrete case study. The overall solution path will remain fixed, while an agent is placed inside the stage where the path cannot be determined in advance.

In this post, we configure a simple agent that only has access to pre-defined tools. If you’d like to upgrade your agent with code execution and web browsing capabilities, check my posts here: Build an LLM Agent That Can Write and Run Code, and How to Give an LLM Agent a Browser.

1. Case Study: LLM-Assisted Hyperparameter Tuning

For the case study, let’s build an LLM application that helps select algorithms and tune hyperparameters for classification problems.

This application works like this: it first receives a labeled dataset and a plain-language modeling request. It then needs to run model experiments and recommend one of the tested configurations. Finally, it summarizes the result.

Naturally, we can break this work into three stages.

1.1 Prepare the Experiment

The purpose of this stage is to translate the modeling request into a brief that contains the objective, evaluation metric, and cross-validation setup.

Since we already know the input, the desired operation to perform, and the expected output, a single LLM call is sufficient for this stage.

1.2 Hyperparameter Tuning

At this stage, we need to run the experiments. Here, we know the goal, i.e., finding the best model and the associated hyperparameters. But we don’t know the exact sequence of actions required to reach it. The next useful experiment should depend on the results observed so far.

As a result, this stage is better handled by an agent, who can dynamically choose classifier configurations, evaluate them, and continue exploring.

1.3 Summarization and Reporting

Finally, we need to summarize the completed run. At this stage, the experiment brief, trial history, and recommendation are already available. Turning them into a structured report is a known operation, so a single LLM call is again sufficient.

As you can see, in our intended LLM application, the overall problem-solving sequence is fixed, with preparation, exploration, and reporting. Within this predefined workflow, we introduce autonomy only at the middle stage to enable adaptive model experimentation.

This way, we combine the clarity of a workflow with the flexibility of agentic experimentation.

In the following, let’s build those three stages.

2. Building the Three-Stage Workflow

The complete application can be expressed in three calls:

experiment_spec = prepare_experiment(
    modeling_request,
    dataset_summary,
)

recommendation, trial_history = await explore_configurations(
    experiment_spec,
    dataset_summary,
)

report = summarize_run(
    experiment_spec,
    trial_history,
    recommendation,
)

Now, let’s unpack the three functions one by one.

2.1 Preparing the Experiment with Structured Output

The first stage converts the modeling request and dataset summary into a compact experiment brief. We use a single structured LLM call to do that.

We need to define the output schema, the LLM instruction, and the prompt builder for this stage. We start with the output schema, which is a Pydantic model:

class ExperimentSpec(BaseModel):
    objective: str
    primary_metric: str = Field(
        description="A valid scikit-learn scoring name"
    )
    cv_folds: int

It has three fields, specifying the information needed by the experimentation agent, i.e., what it should accomplish, how configurations should be evaluated, and how many cross-validation folds to use.

This is the structured output feature of the LLM: it makes sure that the LLM’s output follows this predefined structure, thus greatly simplifying the downstream consumption of the results.

Next, we define the instruction:

PREPARER_INSTRUCTION = """
Create an experiment brief from the modeling request and dataset summary.
"""

Then, we define the builder function to compose the prompt:

def build_preparer_prompt(request: str, dataset_summary: dict) -> str:
    return f"""Modeling request:
{request}

Dataset summary:
{json.dumps(dataset_summary, indent=2)}"""

Finally, we put everything together with a synchronous call to the Responses API:

from openai import AzureOpenAI

llm_client = AzureOpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    azure_endpoint=os.environ["OPENAI_API_BASE"],
    api_version=os.environ["OPENAI_API_VERSION"],
)

def prepare_experiment(
    request: str,
    dataset_summary: dict,
) -> ExperimentSpec:
    response = llm_client.responses.parse(
        model="gpt-5.4",
        reasoning={"effort": "medium"},
        instructions=PREPARER_INSTRUCTION,
        input=build_preparer_prompt(request, data),
        text_format=ExperimentSpec,
    )

    return response.output_parsed

This gives the next stage an explicit experimental contract.

2.2 Building the Experimentation Agent

The purpose of this stage is to find a strong classifier configuration by running experiments.

A single LLM call won’t cut it as the next useful configuration depends on the scores observed so far. Therefore, we use an agent instead to maximize the adaptivity.

We first configure an asynchronous client for powering the agent:

from openai import AsyncAzureOpenAI
from agents import OpenAIResponsesModel

agent_client = AsyncAzureOpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    azure_endpoint=os.environ["OPENAI_API_BASE"],
    api_version=os.environ["OPENAI_API_VERSION"],
)

agent_model = OpenAIResponsesModel(
    model="gpt-5.4",
    openai_client=agent_client,
)

We then define the output schema, instruction, and prompt for the agent:

import json
from pydantic import BaseModel

class AgentRecommendation(BaseModel):
    model_name: str
    hyperparameters_json: str
    rationale: str


AGENT_INSTRUCTION = """
Find a strong classifier for the supplied problem.
"""


def build_agent_prompt(
    experiment_spec: ExperimentSpec,
    dataset_summary: dict,
) -> str:
    return f"""Experiment brief:
{experiment_spec.model_dump_json(indent=2)}

Dataset summary:
{json.dumps(dataset_summary, indent=2)}"""

Next, we define the tool the agent can use to run one experiment:

import json
from agents import function_tool
from sklearn.model_selection import cross_val_score
from sklearn.utils import all_estimators

def build_experiment_tool(
    X,
    y,
    experiment_spec,
    trial_history,
):
    classifiers = dict(
        all_estimators(type_filter="classifier")
    )

    @function_tool
    def run_experiment(
        model_name: str,
        hyperparameters_json: str,
    ) -> str:
        """Evaluate one scikit-learn classifier configuration."""
        parameters = json.loads(hyperparameters_json)
        classifier = classifiers[model_name](**parameters)

        scores = cross_val_score(
            classifier,
            X,
            y,
            cv=experiment_spec.cv_folds,
            scoring=experiment_spec.primary_metric,
        )

        result = {
            "model_name": model_name,
            "hyperparameters": parameters,
            "mean_score": round(float(scores.mean()), 4),
        }

        trial_history.append(result)
        return json.dumps(result)

    return run_experiment

This tool is intentionally small, and it simply evaluates the classifier configuration supplied by the agent. Also, the agent can choose classifiers from scikit-learn’s classifier registry and provide constructor arguments as JSON.

Now we can flesh out the agentic stage in full:

# pip install openai-agents
from agents import Agent, ModelSettings, Runner

async def explore_configurations(
    X,
    y,
    experiment_spec,
    dataset_summary,
):
    trial_history = []

    run_experiment = build_experiment_tool(
        X,
        y,
        experiment_spec,
        trial_history,
    )

    agent = Agent(
        name="Model selection agent",
        instructions=AGENT_INSTRUCTION,
        model=agent_model,
        model_settings=ModelSettings(
            reasoning={"effort": "medium"},
            parallel_tool_calls=True,
        ),
        tools=[run_experiment],
        output_type=AgentRecommendation,
    )

    result = await Runner.run(
        agent,
        build_agent_prompt(
            experiment_spec,
            dataset_summary,
        ),
        max_turns=10,
    )

    return result.final_output, trial_history

Note that we set parallel_tool_calls=True. This allows the agent to request several configurations concurrently.

This is the only autonomous part of the workflow.

2.3 Summarizing the Run

The final stage needs to turn the completed run into a concise report. This is a closed-ended task, therefore a single LLM call is sufficient.

As usual, we start by defining the output schema:

from pydantic import BaseModel

class ModelSelectionReport(BaseModel):
    selected_model: str
    selected_hyperparameters: str
    mean_cv_score: float
    summary: str

Then, we define the instruction and prompt builder:

REPORTER_INSTRUCTION = """
Summarize the completed model-selection run.
"""

def build_reporter_prompt(
    experiment_spec: ExperimentSpec,
    trial_history: list[dict],
    agent_recommendation: AgentRecommendation,
) -> str:
    return f"""Experiment brief:
{experiment_spec.model_dump_json(indent=2)}

Completed trials:
{json.dumps(trial_history, indent=2)}

Agent recommendation:
{agent_recommendation.model_dump_json(indent=2)}"""

Finally, we wrap the LLM call in a function:

def summarize_run(
    experiment_spec: ExperimentSpec,
    trial_history: list[dict],
    agent_recommendation: AgentRecommendation,
) -> ModelSelectionReport:
    response = llm_client.responses.parse(
        model="gpt-5.4",
        reasoning={"effort": "medium"},
        instructions=REPORTER_INSTRUCTION,
        input=build_reporter_prompt(
            experiment_spec,
            trial_history,
            agent_recommendation,
        ),
        text_format=ModelSelectionReport,
    )

    return response.output_parsed

3. Testing the Workflow on Handwritten Digit Classification

To test what we have built, we use scikit-learn’s built-in handwritten digits dataset (CC BY 4.0):

from sklearn.datasets import load_digits

digits = load_digits()

X = digits.data
y = digits.target

dataset_summary = {
    "n_samples": X.shape[0],
    "n_features": X.shape[1],
    "n_classes": len(set(y)),
}

Here is our modeling request:

modeling_request = """
Build a classifier for handwritten digit images.
Use cross-validation to compare candidate models.
Recommend a strong configuration.
"""

Now we can run the three-stage workflow:

experiment_spec = prepare_experiment(
    modeling_request,
    dataset_summary,
)

recommendation, trial_history = await explore_configurations(
    X,
    y,
    experiment_spec,
    dataset_summary,
)

report = summarize_run(
    experiment_spec,
    trial_history,
    recommendation,
)

Here is what the first LLM call produced in experiment_spec:

{
    "objective": "Build and evaluate a classifier for handwritten digit images.",
    "primary_metric": "accuracy",
    "cv_folds": 5,
}

The agent then uses the experiment tool to explore model configurations. In my run, the total length of trial_history is 19, here are the first 5 trials:

[
    {
        "model_name": "LogisticRegression",
        "hyperparameters": {
            "max_iter": 1000
        },
        "mean_score": 0.9132
    },
    {
        "model_name": "RandomForestClassifier",
        "hyperparameters": {
            "n_estimators": 100,
            "max_depth": null
        },
        "mean_score": 0.9382
    },
    {
        "model_name": "SVC",
        "hyperparameters": {
            "C": 1,
            "kernel": "rbf",
            "gamma": "scale"
        },
        "mean_score": 0.9638
    },
    {
        "model_name": "KNeighborsClassifier",
        "hyperparameters": {
            "n_neighbors": 3
        },
        "mean_score": 0.9661
    },
    {
        "model_name": "SVC",
        "hyperparameters": {
            "C": 10,
            "kernel": "rbf",
            "gamma": "scale"
        },
        "mean_score": 0.975
    }
]

Finally, the last LLM call summarizes the completed run in report:

{
    "selected_model": "SVC",
    "selected_hyperparameters": '{"C": 10, "kernel": "rbf", "gamma": "scale"}',
    "mean_cv_score": 0.975,
    "summary": "The SVC configuration achieved the strongest cross-validation score among the tested candidates and is recommended as the final classifier.",
}

4. When to Use This Pattern

When building your next LLM application, you shouldn’t ask yourself whether the whole application should be a workflow or an agent.

A better question is: where does the application need autonomy?

If a stage has a known operation, use a structured LLM call.

If a stage has a clear goal, but the next action depends on what the system observes, go with an agent.

Decompose the solution path into stages, and place autonomy only where the path needs to be discovered. That’s how you maintain the clarity and control of a workflow while using agentic flexibility for more effective problem solving.