Agent memory: what to remember, where to put it, when to forget

A model with no memory is a very good stranger. It answers each request well, then forgets you exist. That is fine for a single question. It stops being fine once you build an agent that works with the same data stewards every day, reviews the same entities week after week, and is expected to learn that one team always trusts the Salesforce address over the ERP one.

At Data Color AI I build agents that sit inside master data management workflows. Memory is the part of those systems I have rewritten the most. The first version saved everything and retrieved too much. The second saved almost nothing. What works is less about the vector database and more about three policies: what gets written, what gets retrieved, and what gets forgotten.

Four kinds of memory

The cognitive science terms map onto agents better than I expected, and they are useful because each type wants a different storage layer and a different lifetime.

• Working memory is what the model can see on this turn: the system prompt, the conversation, tool results, any scratchpad. It lives in the context window and dies when the run ends.
• Episodic memory is a record of what happened. “On June 12 the steward approved merging ACME Corp and Acme Inc.” It is useful for continuity and for learning from past outcomes.
• Semantic memory is distilled facts. “This steward prefers the Salesforce address.” “Customer 4471 is a subsidiary of 1022.” No timestamp story, just what is true now.
• Procedural memory is how to do things: the system prompt, playbooks, tool descriptions, skill files. It changes rarely and should be versioned like code.

The mistake I made early was treating all four as “stuff to embed and search.” Procedures do not belong in a similarity search. If the agent needs to know how to run a merge review, that belongs in its instructions every time, not in a store that retrieves it only when the query happens to be phrased the right way.

Where each one lives

In practice I end up with four storage layers, and a clear rule for which memory goes where.

1. The context window for working memory. It is the most expensive and the least durable layer, so nothing should be there by default.
2. A checkpoint store for the state of a thread: messages, intermediate state, which node the graph was on. This is what lets a conversation resume after a human approval that took two days.
3. A key-value profile for small, structured facts about a user or entity. Preferences, role, the systems they own. Read by key, not by similarity. Cheap and exact.
4. A vector index for free-form semantic and episodic memories where you do not know the key in advance. “Anything relevant about how this team handles duplicate suppliers?”

The profile is underrated. A large share of what agents “remember” in production is structured: a name, a preferred source system, a region. Putting that in a vector store means paying for embeddings and getting fuzzy retrieval for something that has an exact answer.

Short-term and long-term in LangGraph

LangGraph makes this split explicit, which is one reason I use it. A checkpointer saves graph state after every step, keyed by thread_id. That is short-term memory: everything about one conversation or one job, including the ability to pause for a human and resume later. A Store is separate. It holds JSON documents under namespaces such as ("user", user_id) or ("entity", entity_id), optionally with a vector index, and every thread can read from it.

Keeping these apart matters. Checkpoints grow with every step and are mostly noise after the thread ends. The store should only contain things you would be comfortable showing the agent in any future conversation.

Write policy: what is worth remembering

This is the hard part, and it is a product decision more than an engineering one. My current rule is that a memory has to pass three tests:

• Will it change a future decision? “The user said thanks” will not. “The user rejects merges when tax IDs differ, even if names match” will.
• Is it stable? A preference or a relationship is. The status of a ticket is not; read that from the source system instead.
• Is the agent allowed to keep it? More on privacy below.

Then there is the question of who decides. There are two approaches, and I use both.

For the AI Data Steward, the agent saves explicit instructions immediately (“always keep the D&B DUNS number”), and a background job extracts softer patterns from approval history overnight.

from pydantic import BaseModel, Field
from langchain_core.tools import tool

class Memory(BaseModel):
    subject: str = Field(description="Who or what this is about, e.g. 'user:priya' or 'entity:4471'")
    fact: str = Field(description="One self-contained statement, written so it makes sense out of context")
    kind: str = Field(description="'preference' | 'relationship' | 'instruction' | 'outcome'")
    importance: float = Field(ge=0, le=1, description="How much this should change future decisions")
    supersedes: str | None = Field(default=None, description="ID of a memory this one replaces")

@tool(args_schema=Memory)
def save_memory(subject, fact, kind, importance, supersedes=None):
    """Save a durable fact that will change how you act in future sessions.
    Do not save small talk, transient status, or anything readable from a source system."""
    ns = tuple(subject.split(":", 1))
    if supersedes:
        store.put(ns, supersedes, {**store.get(ns, supersedes).value, "valid_to": now()})
    store.put(ns, new_id(), {"fact": fact, "kind": kind, "importance": importance, "valid_from": now()})
    return "saved"

Two details in that tool description do most of the work: the fact must be self-contained, and the description tells the model what not to save. Without the second one, agents save everything.

Retrieval: similarity is not enough

Pure vector similarity has a known problem. It returns things that sound related, not things that matter. I score candidates on three signals, an approach popularized by the generative agents paper and still a good default:

• Similarity to the current query.
• Recency, with exponential decay since the memory was last used.
• Importance, assigned at write time.

import math

def score(mem, sim, now, half_life_days=30, w=(0.6, 0.2, 0.2)):
    age_days = (now - mem["last_used"]).total_seconds() / 86400
    recency = math.exp(-math.log(2) * age_days / half_life_days)
    return w[0] * sim + w[1] * recency + w[2] * mem["importance"]

def recall(query, ns, k=3, threshold=0.5):
    hits = store.search(ns, query=query, limit=20, filter={"valid_to": None})
    ranked = sorted(hits, key=lambda h: score(h.value, h.score, now()), reverse=True)
    return [h for h in ranked if score(h.value, h.score, now()) >= threshold][:k]

Two things in that code matter more than the weights. The metadata filter valid_to: None excludes superseded facts before ranking. And the threshold means the agent can retrieve nothing. An empty result is much better than three weak memories that pull the model off course.

Updates: supersede, don't append

Facts change. The steward for a region moves teams, and a customer gets acquired. If memory is append-only, retrieval eventually returns both the old and new fact, and the model picks one, often the wrong one because the old fact has more supporting memories around it.

On every write, I search for existing memories about the same subject and ask a small model a narrow question: does the new fact duplicate, update, or contradict any of these? Duplicates are dropped. Updates and contradictions close the old record with a valid_to timestamp. In an MDM context this is familiar ground, because it is the same bitemporal thinking data teams already apply to golden records.

Forgetting on purpose

Memory that only grows gets worse over time. Retrieval gets noisier, costs go up, and stale facts accumulate. I use three forgetting mechanisms:

• Decay. Recency weighting already pushes unused memories down. Memories that have not been retrieved in a long time and have low importance get archived.
• TTL by kind. Outcomes (“merge approved on June 12”) expire after a few months. Instructions do not expire until someone changes them.
• User control. People can see and delete what the agent remembers about them. This matters for trust, and it is a legal requirement in many places.

Failure modes I have hit

Stale facts

The agent confidently uses a preference from six months ago. The fix is supersession plus decay, and for anything that has a source system, reading the source instead of remembering it.

Memory poisoning

If an agent saves content from documents or tool outputs, then a malicious or simply wrong document can plant a “fact” that shapes every future session. I only allow durable memories that come from the user or from verified outcomes. I never save text that came from a retrieved document, and every memory records its provenance.

Over-retrieval

Retrieving ten memories “just in case” fills the context with loosely related facts, and the model treats everything in context as relevant. It will try to use them. A threshold and a small k fix most of this.

A good memory system is judged by what it leaves out.

Start with the profile and the checkpointer. They cover most of what users experience as “it remembers me.” Add a vector store for episodic and semantic memory only when you have a clear write policy, a supersession rule and a way to forget. Without those three, memory makes the agent worse the longer it runs.