Not everything an agent sees is worth remembering. Store too little and it forgets what mattered; store everything and the memory fills with noise that drowns the signal at retrieval time.

The extraction step — deciding what to keep and distilling it into clean, self-contained facts — is the most consequential and most overlooked part of a memory system.

Memory begins with a decision

Of everything that just happened, what's worth keeping? This is the first and most important question a memory system answers, and it's easy to get wrong in both directions. Store too little and the agent forgets the thing that mattered. Store everything and the memory fills with noise, because every irrelevant memory is a candidate to surface at the wrong moment and drown the useful one.

What deserves to be remembered

  • Stable facts about the user — preferences, context, recurring needs that will matter in future sessions.
  • Decisions and outcomes — what was chosen, what worked, what failed, so the agent doesn't relitigate settled ground.
  • Corrections — when a human fixes the agent, that correction is high-value memory: it must stick.
  • Durable state — the status of ongoing work, entities the agent tracks, anything with a life beyond one turn.

What to leave out

Transient chatter, one-off details with no future relevance, and anything that will be stale by next session are noise. The discipline of not storing them is as important as the discipline of storing what matters — a memory full of irrelevant fragments retrieves worse than a lean one.

Good memory is curated, not recorded.

Free Agent Memory Quick-Start — the four kinds of memory and the whole loop (store, retrieve, reflect, forget) on a few pages. Download it free.

Turn experience into clean facts

Raw conversation is messy; stored memories should be clean. Extraction usually distills an interaction into concise, self-contained statements — "user prefers X," "project deadline is Y" — rather than dumping the transcript. A common pattern is to use the model itself to summarize what's worth remembering, producing tidy semantic facts from noisy episodic experience.

memories = extract(conversation)
# -> [{'content': 'prefers email over phone',
#      'subject': 'user_1837', 'type': 'semantic'},
#     {'content': 'had billing issue, resolved by refund',
#      'subject': 'user_1837', 'type': 'episodic'}]
store.add(memories)

Enter fullscreen mode Exit fullscreen mode

Self-contained is the key property: a memory that only makes sense with its original context will confuse retrieval later.

When to extract

There's also a question of timing. The common choices are at the end of a session, when the whole interaction can be distilled at once, or incrementally as significant things happen, so nothing is lost if a session is abandoned. End-of-session is simpler and cheaper but risks losing an interaction that never cleanly ends; incremental captures more reliably at the cost of running more often. Many systems do both.

Going deeper? AI Agent Memory: The Complete Guide is the full reference — 41 pages, 15 chapters, 5 appendices, with a worked support-agent example and a 30-day adoption path. Get the guide.

FAQ

What should an AI agent store in memory?

Stable facts about the user (preferences, context), decisions and outcomes, corrections from humans, and durable state about ongoing work. The test is whether it will matter in a future session.

What should an agent NOT store?

Transient chatter, one-off details with no future relevance, and anything stale by next session. Not storing noise is as important as storing signal — irrelevant memories surface at the wrong time and degrade retrieval.

How does memory extraction work?

It distills a messy interaction into concise, self-contained facts — 'user prefers X' — rather than storing the raw transcript. A common approach uses the model itself to summarize what's worth remembering into clean semantic facts.

When should extraction run?

At the end of a session (simpler, cheaper) or incrementally as important things happen (more reliable, more frequent). Many systems do both — capturing clearly important facts immediately and distilling the rest at the end.

Why is storing less sometimes better?

Because every stored memory is a candidate to surface at retrieval time. A store full of irrelevant fragments retrieves worse than a lean one, burying the memory that mattered under ones that didn't.