The first time you watch an agent fail because of memory, it’s almost funny. A support assistant issues a refund on Tuesday. On Thursday the same customer writes back, and the agent cheerfully asks for an order number it was given two days earlier. Nothing crashed. No error appeared in the logs. The agent simply had no idea the earlier conversation had ever happened.
That gap is where a lot of AI agent projects quietly stall. Prompting gets you a competent single turn. Memory is what turns a series of turns into something that behaves like a colleague who’s been around long enough to know the account. Unlike prompting, memory is an engineering problem: storage, retrieval, ranking, eviction, and a set of failure modes that never show up in a demo.
Three kinds of memory, three different problems
People say “AI agent memory” as if it were a single component. It isn’t, and merging the pieces is how teams end up dumping everything into a vector database and wondering why the agent still repeats itself.
- Working memory: whatever sits in the context window right now. The system prompt, the last several turns, tool results, scratch notes. Fast, expensive per token, and gone the moment the request finishes.
- Episodic memory: records of specific past interactions. On 12 March this user asked about invoice 4471, and we sent a corrected PDF the same afternoon.
- Semantic and procedural memory: durable facts and know-how. The customer is on the enterprise plan. Refunds above £500 need manager sign-off. This endpoint takes a POST, not a GET.
Each has a different access pattern. Working memory is read on every single turn and needs to be cheap to assemble. Episodic memory is written constantly but read rarely and selectively. Procedural memory changes slowly, and when it changes it needs to be right, because a wrong fact in a system prompt propagates into every answer the agent gives.
The context window is not memory, even when it’s enormous
Modern models accept 200,000 tokens or more. That sounds like it removes the problem. It doesn’t, for three reasons that show up in production within a week.
The first is cost and latency. A 200k-token prompt at typical API pricing runs roughly thirty times the cost of a 6k-token prompt, and the time-to-first-token grows accordingly. If your agent handles 40,000 conversations a month, that difference is a budget line, not a rounding error.
The second is attention. Recall degrades for information buried in the middle of a long context, which is exactly where the detail you needed from turn three tends to land.
The third is scope. A context window dies with the request. Memory is supposed to outlive it, follow the user across sessions, and stay strictly separated between one customer and the next. Those are database properties, not prompt properties.
How teams actually implement long-term memory
Vector retrieval, the default choice
Chunk past turns, embed them, store them, pull the top k by cosine similarity at query time. This works acceptably for FAQ-style support and badly for anything temporal or aggregative. Ask “what changed in this account last quarter?” and similarity search hands back five chunks that mention the account, not a timeline.
Explicit state and graphs
Graph-based orchestration frameworks treat memory as declared state rather than a prompt-stuffing exercise. LangGraph’s approach to state, control and memory is worth studying even if you never use the library, because it forces you to name what persists between steps and what gets checkpointed per thread. Once those boundaries are explicit, persistence, replay and human review all become tractable.
Agents that edit their own memory
A third camp hands the keys to the model. Letta, the open-source framework built around self-editing memory, gives the agent a small set of memory blocks it can rewrite, plus a paging mechanism for pulling older material back into context when it decides it needs it. It’s a genuinely different design: the agent, not your retrieval heuristic, decides what deserves to be remembered.
Where memory quietly falls apart
Most memory bugs are silent. The agent doesn’t error out, it just gets subtly worse, and by the time anyone notices you’ve got six weeks of bad records to clean up. The usual suspects:
- Stale facts with no timestamp. The customer upgraded to the pro plan in April. In September the agent still quotes enterprise pricing because that chunk ranked highest on similarity.
- Contradictions with no resolution rule. Two memories disagree and retrieval picks one at random. Recency, confidence and source authority all need to be part of the ranking, not an afterthought.
- Memory poisoning. Anything the agent writes to memory becomes trusted input later. A malicious or careless instruction buried in a document can end up as a durable belief. Memory writes deserve the same scrutiny as user input.
- Unbounded growth. Retrieval quality drops as the store fills with near-duplicates, and every query gets slower and more expensive.
These are the same structural weaknesses that make agents break down in production while demos look flawless. A scripted demo never runs long enough for staleness to bite.
Design rules that survive contact with real users
Six habits separate memory systems that hold up from ones that become a liability.
- Write deliberately. Summarise a completed task into two or three structured facts rather than appending the whole transcript. Every stored token is a future retrieval candidate and a future liability.
- Timestamp and version everything. Memory without a date is a rumour. Store when it was written and when the fact became true.
- Scope hard. Per-user and per-tenant partitions enforced at the query layer, not by filtering after retrieval.
- Keep an audit trail a human can read. When an agent does something odd, you need to see which memories it was holding, in plain language.
- Budget the retrieval. Cap the tokens spent on memory per turn. A 2,000-token memory budget with good ranking beats a 30,000-token dump almost every time.
- Split structured from narrative. Account tier, preferences and permissions belong in fields you can query exactly. Free-text summaries belong in retrieval.
Test memory the way you’d test a database
Prompt evaluations won’t catch memory regressions. You need scenarios that unfold over time. Build a set of fifty scripted multi-session conversations, each with a known ground truth, and check three things: does the agent recall the fact when it matters, does it correctly ignore facts that are out of scope, and does it resolve conflicts in favour of the more recent, more authoritative record.
Add an adversarial set too. Feed it a document containing an embedded instruction and confirm nothing durable gets written. Run the whole suite after every change to your extraction or ranking logic, because those changes alter behaviour far more than swapping models usually does.
Decide what the agent should forget
Forgetting is the part everyone skips. A memory store with no expiry policy becomes an archaeological dig, and the agent’s answers get slower and vaguer as it goes. Write down the retention rules before launch: session transcripts kept for 30 days, extracted facts for as long as the account is active, a temporary preference that expires in a fortnight, anything derived from a document deleted when the document is.
It helps to think of memory as one phase of a longer arc rather than a permanent fixture, which is roughly how the AI agent lifecycle from first prompt to graceful retirement frames it. Agents get scoped down, merged, replaced. When that happens, the memory has to be migrated, archived or destroyed on purpose. Teams that skip the forgetting policy end up explaining to a regulator, or a very annoyed customer, why an assistant still remembers something it was never supposed to keep.