Most “agent” demos look impressive until you point them at real data. The moment a model has to look up a policy document, hit an internal API, and remember what the user said three turns ago, the happy path collapses. LlamaIndex agents exist to close that gap, because LlamaIndex started life as a retrieval framework and only later grew an agent layer on top of it.
That ordering matters. Instead of bolting retrieval onto an agent, you get an agent that sits directly on top of indexes, query engines, and document stores you probably already have. Here is how the pieces fit together, and where they tend to break.
What LlamaIndex means by “agent”
The framework splits the concept in two. An AgentWorker holds the reasoning loop, deciding what to do next given the conversation so far. An AgentRunner holds the state, the memory, the tools, and the callbacks, and it drives the worker. You can swap a worker for a different reasoning strategy without touching the surrounding plumbing.
If you have not worked with the framework before, the underlying data framework that makes RAG work is worth reading first. Agents are the thinnest layer on top of it, and understanding indexes and query engines makes everything below much easier to follow.
Two worker types cover most use cases. FunctionAgent uses native function calling and is the default choice for models that support it, which today is most of them. ReActAgent falls back to the older text-based reasoning-and-acting pattern, useful for local models or providers without tool-calling APIs. Both expose the same run() interface, so switching later is a one-line change.
The parts you actually touch
Tools are the whole ballgame
A LlamaIndex agent is only as good as its tool descriptions. Every tool is just a Python function with a docstring, and the model reads that docstring to decide when to call it. Vague descriptions produce confident wrong answers. “Searches documents” tells the model almost nothing. “Search the 2024 HR policy handbook for leave entitlements; input is a natural language question” gives it something to work with.
The framework also accepts FunctionTool wrappers for functions you do not want the model to inspect directly, which lets you override the name, description, and schema by hand. That is worth doing for anything that touches money or production data.
Memory and state
Memory in LlamaIndex is a token buffer plus optional summarisation, passed into the runner at construction time. The default keeps a rolling window. For long support conversations you will want to add a summary buffer or persist state to a store, otherwise the agent quietly forgets the customer’s account number halfway through the ticket.
State objects are the other half of this. Anything you want to survive between steps, like a shopping cart or a resolved entity, lives in a typed state dict that gets passed to each tool. It is a small feature that saves an enormous amount of prompt engineering.
A minimal agent, end to end
Current versions make this genuinely short. Define a function, add a docstring, hand it to the agent.
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI
def get_weather(city: str) -> str:
"""Return current weather for a city name."""
return f"18C and cloudy in {city}"
agent = FunctionAgent(
tools=[get_weather],
llm=OpenAI(model="gpt-4o-mini"),
system_prompt="You are a concise weather assistant.",
)
response = await agent.run(user_msg="Do I need a jacket in Lisbon?")
That call returns a result object with the final response, plus the full list of tool calls made along the way. Log that list in development. Reading the tool-call trace is the fastest way to understand why an agent hallucinated.
Turning your index into a tool
This is where LlamaIndex pulls ahead of general-purpose agent libraries. Any query engine can become a tool in one wrapper.
QueryEngineTool.from_defaults(query_engine=index.as_query_engine(), name="contracts", description="...") gives the agent a searchable window into a vector index. Point several of these at different indexes, one for contracts, one for support tickets, one for product docs, and the agent routes questions to the right corpus on its own. Sub-question engines and recursive retrievers wrap the same way, so you can give an agent a tool that decomposes complex multi-document questions without changing the agent code at all.
The same pattern works for non-retrieval actions. Third-party integrations are handled well by giving AI agents real tools without the integration headache, and those toolkits convert into LlamaIndex tools cleanly when you would rather not write forty API wrappers by hand.
Multi-agent setups and handoffs
One agent with twelve tools almost always underperforms four agents with three tools each. LlamaIndex handles this with AgentWorkflow, which lets agents hand control to one another and return a result. A typical build has a triage agent that reads the request, a research agent with index tools, a billing agent with API access, and a writer that assembles the reply.
The handoff happens through tool calls. Each agent gets the others exposed as tools, so “transfer to billing” is just a function call, and the workflow tracks who currently owns the conversation. You can also go lower level with Workflow events if you need branching, loops, or human-approval steps that do not fit the handoff model.
Where LlamaIndex agents fall over
The failure modes are consistent enough to plan for.
- Too many tools. Beyond roughly ten to fifteen, selection accuracy drops noticeably. Split into sub-agents instead of adding another tool.
- Silent loop costs. A misconfigured ReAct agent can call the same tool five times in a row. Set a max iteration cap and check the trace.
- Context blowout. Retrieved chunks plus memory plus tool schemas can eat a 128k window faster than expected. Prune retrieved nodes before they reach the agent.
- Untested prompts. Tool descriptions behave like prompts and break the same way. Version them and rerun an eval set after any edit.
- No observability. Callbacks exist for a reason. Wire them to your tracing tool before launch, not after the first incident.
How to choose between agent frameworks
LlamaIndex agents are the right call when retrieval over your own documents is the centre of gravity. If you are mostly orchestrating typed business logic, type-safe AI agents in Python built the FastAPI way give you stronger validation guarantees with less overhead. On the cloud side, Google Vertex AI Agent Builder and Databricks Mosaic AI Agents bring managed infrastructure and governance that self-hosted LlamaIndex does not attempt.
Plenty of teams mix them. LlamaIndex handles the document layer and exposes an API endpoint; a stricter orchestration framework calls that endpoint as one tool among many.
What a production setup looks like
Strip away the tutorials and a shipped LlamaIndex agent tends to have the same shape. Tools are thin, typed, and individually unit-tested. Retrieval is tuned separately from the agent, because a poorly chunked index cannot be rescued by a better prompt. Memory is capped and summarised. Every run writes a trace with tool names, latencies, and token counts. An offline eval set of 50 to 100 real user questions runs on every prompt change, comparing final answers and, more importantly, whether the correct tools were chosen.
The eval set is the part teams skip and later regret. Agent quality is mostly a function of tool selection, and tool selection is invisible unless you measure it directly. Start with twenty hard questions from your own support inbox. Watch which tools fire, fix the descriptions that misfire, and rerun. That loop will improve a LlamaIndex agent faster than any model upgrade.