Most people who open Dify for the first time do the same thing. They drop a Start node, an LLM node and an Answer node onto the canvas, wire them together, and have a working chatbot in about eleven minutes. Then they try to make it do something that matters, and discover the wiring was never the interesting part.
Dify is an open-source platform for building LLM applications: chat assistants, retrieval pipelines, and AI agents. It runs on your own servers, talks to almost any model provider, and exports every app as a JSON file you can commit to git. Those three facts explain why it keeps turning up in production stacks long after the flashier demo tools have been forgotten.
What Dify actually means when it says “agent”
Inside Dify, an agent is one node type. You place it on the canvas, pick a strategy, and attach the tools it is permitted to touch: built-in web search, a custom API tool defined by an OpenAPI schema, or something pulled from the plugin marketplace. Give it a prompt, set a maximum iteration count, and it loops through reasoning and tool calls until it produces an answer.
Two strategies ship out of the box. Function Calling hands the model a structured tool schema and lets it decide when to invoke each one, which works well with modern models that support native tool use. ReAct is the older reasoning-and-acting loop where the model writes out its thoughts before each call. If the vocabulary around these approaches feels opaque, there is a decent primer on AI terms that trip people up worth reading before you pick one.
Everything around the agent node is deliberately deterministic: HTTP requests, conditional branches, loops over lists, Python or JavaScript code steps, variable aggregation. You decide how much autonomy to hand over at each point rather than betting the whole pipeline on a single prompt behaving itself.
When the agent node earns its keep
Reach for it when the number of plausible next actions genuinely changes at runtime. Support triage is the classic case: an incoming message might need a knowledge base lookup, an order status API call, or a refund tool, and you cannot know which until you read it. Inventory questions and internal IT helpdesks behave the same way.
Skip it when the sequence is fixed. If every request takes the same four steps in the same order, a plain chain of nodes will be faster, cheaper and far easier to debug. Plenty of teams add an autonomous agent to a process that never needed one.
Why the workflow canvas usually beats full autonomy
Consider a contract review app. A four-step chain reads the document, extracts the clauses you care about into structured JSON, runs a code node to check them against your thresholds, then routes anything unusual to a human reviewer. That chain returns the same shape of answer every time and takes about nine seconds. Swap the middle for an agent and you get more flexibility, a wider latency range, and a new class of failure where the model decides to skip a step.
Dify’s iteration node is the feature that quietly does the most work here. Feed it a list of 40 line items from an invoice and it runs the same sub-workflow against each one, collecting results you can then aggregate. Agent frameworks elsewhere make you write that loop yourself and handle the partial failures.
Retrieval is where most Dify AI agents fail quietly
Knowledge base settings get treated as an afterthought, and that is usually the real problem. Dify lets you tune chunk size, chunk overlap, separators, indexing method and retrieval mode. The defaults are reasonable for a FAQ page and terrible for a 200-page technical manual.
A few settings that move the needle:
- Chunk size around 500–800 tokens with 10–15% overlap suits dense documentation better than the larger defaults, which bury answers inside paragraphs of context.
- Hybrid search combines keyword and vector matching. Turn it on when users search for part numbers, error codes or product names that embeddings handle poorly.
- A rerank model costs one extra API call per query and frequently doubles retrieval precision. Worth testing before you touch the prompt.
- Metadata filters let you scope retrieval by department, product line or document date, which is often the cleanest fix for a knowledge base that has grown past a few thousand chunks.
If retrieval returns the wrong five chunks, no amount of prompt engineering recovers. Fix the data layer first, then complain about the model.
Self-hosting versus the cloud plan
The Docker Compose deployment spins up PostgreSQL, Redis, a vector store such as Weaviate or Qdrant, the API and worker containers, plus a plugin daemon and a sandboxed container that executes code nodes. That is a lot of moving parts, but a 4 vCPU / 8GB instance handles a small team comfortably.
Self-hosting makes sense when data residency is non-negotiable, when you need to point at a local model through Ollama or vLLM, or when per-seat cloud pricing starts to look silly against your usage. Upgrades are the part people underestimate: back up the database, read the release notes for migrations, and never jump more than one minor version at a time.
Plugins, and where the marketplace stops being enough
Dify’s marketplace covers hundreds of tools and model providers, from Google Search to Slack to a long list of regional LLM vendors. For a prototype, that is plenty. In production, pin plugin versions and treat third-party plugins as dependencies that can rot. Anything business-critical is better wrapped as a custom tool with an OpenAPI schema you control, because then the contract is yours.
Debugging a Dify agent without guessing
Every workflow run stores per-node inputs and outputs, which sounds mundane until you have spent an afternoon in a framework that gives you a stack trace and nothing else. You can replay a single node, inspect the exact string sent to the model, and compare two runs side by side. Annotation replies let you hardcode known-good answers for questions that should never reach the LLM at all.
For anything customer-facing, wire the logs into an observability tool like Langfuse or LangSmith. The habits that separate working agents from expensive toys are covered in this AI agent development roadmap, and most of them apply directly to a Dify build.
When Dify is the wrong choice
RPA-heavy enterprises with thousands of existing bot workflows often get more from something like Automation Anywhere’s AI Agent Studio, which plugs into the orchestration layer they already pay for. Real-time voice agents need a different stack entirely, and the trade-offs are laid out in this look at Pipecat and voice agent frameworks. Latency budgets under a second do not survive a standard request-response workflow.
Cost is the other consideration. Dify’s free self-hosted tier has licence conditions around multi-tenant resale, and the cloud plans meter message credits in ways that surprise people mid-month. This breakdown of what free AI agent tiers really include is a useful reality check before you commit.
A first agent that actually ships
If you want something live this week rather than a demo nobody uses, keep the scope this tight:
- Start with one workflow and one job. “Answer questions about our returns policy and file a return” is a product; “help customers” is not.
- Build the deterministic version first. Chain the nodes, get the happy path working, then convert only the branching step into an agent node.
- Cap the iteration count at three or four. Agents that loop more than that are usually confused, and you are paying for the confusion.
- Add a code node to validate every structured output before it reaches anything downstream.
- Log everything from day one, including the retrieval chunks returned, not just the final answer.
- Write twenty test questions with known correct answers and rerun them after every prompt change.
The maintenance nobody budgets for
A Dify agent is not finished when it goes live. Model providers deprecate versions on their own schedule, sometimes within a year. The API your custom tool wraps will change a field name. Knowledge bases drift as old documents become wrong. Each of those events can quietly degrade quality without throwing a single error.
Teams that keep agents healthy tend to do three unglamorous things. They pin model versions deliberately rather than always chasing the newest release. They keep the twenty-question test set and run it before any change goes out. And they review a sample of real conversations every week, because a workflow returning confident nonsense looks identical to one returning good answers until you read the transcripts. Do that, and a Dify build holds up for years. Skip it, and you have an expensive chat window with a plugin list nobody dares touch.