Agent Memory Is a Systems Workload: What SelfMem Changes—and What It Does Not

Three recent papers should change how teams design agent memory.

SelfMem, submitted July 4, shows that an agent can outperform fixed retrieval and summarization strategies by learning how to manage its own memory workspace.

Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads, submitted June 4, profiles ten memory systems and shows that their dominant production cost often sits on the write path, not the query path.

From Untrusted Input to Trusted Memory, also submitted in June and presented at an ICML 2026 workshop, shows that aggressive write and retrieval behavior can make persistent memory easier to poison.

Together they expose the real design problem:

Agent memory is an optimization problem, a systems workload, and a security boundary at the same time.

Optimizing only recall produces an expensive memory system. Optimizing only latency produces a forgetful one. Allowing a model to optimize writes without provenance and promotion controls produces a durable attack surface.

The right production architecture keeps an immutable evidence log, lets the model propose task-sensitive memory, and places deterministic eligibility, promotion, retention, and recall controls around it.

This review distinguishes each paper’s reported result from our production inference. SelfMem and the workload characterization are recent preprints; benchmark improvements are not guarantees about a different model, workload, or deployment.

SelfMem’s strongest idea is not self-modification

SelfMem is often summarized as “an agent that writes its own memory.” That misses the more useful architectural choice.

Its source of truth is an immutable conversation transcript. The agent accesses that transcript through read-only tools, maintains a separate mutable workspace, and can:

• inspect source turns;
• write, replace, merge, refine, or archive memory;
• review proposed or existing entries for missing support, staleness, contradiction, or retrievability;
• create internal probes to test whether the memory organization can answer plausible future questions;
• retrieve exact evidence from the transcript when compact memory is insufficient.

The model chooses the concrete strategy, but the framework supplies the tools, constraints, audit fields, and feedback channels.

That is the pattern worth keeping:

immutable history
  -> read-only inspection
  -> model-managed workspace
  -> review and revision
  -> source-grounded retrieval
  -> answer

The mutable memory is not the evidence. It is an index, summary, and working theory over the evidence.

This distinction prevents one of the most common memory mistakes: asking a lossy summary to act as both compressed context and authoritative history.

What the SelfMem result supports

SelfMem evaluates on BEAM: 100 long conversations and 2,000 questions at 100K, 500K, and 1M-token scales. Under the paper’s shared harness, answer model, and judging protocol, it reports the highest score at all three lengths:

The strongest baseline was simple RAG in each row. Some structured memory systems cost much more under the same protocol while scoring lower; at 1M tokens, for example, the paper reports $18.830 for Mem0 and $2.004 for SelfMem.

Those numbers are evidence for task-sensitive memory management, not a universal product comparison. The authors note several limits:

• evaluation is concentrated on one benchmark and fixed judging protocol;
• iterative strategy refinement is studied at 100K rather than all three scales;
• additional model families and deployment settings remain to be tested;
• specialized production implementations may have different efficiency profiles.

The defensible lesson is:

Fixed chunking, fixed summaries, and fixed profile schemas leave performance on the table when future memory demands vary. Giving an agent constrained memory tools can discover better organization and retrieval behavior.

The dangerous extrapolation is:

Therefore the agent should be allowed to decide what becomes trusted persistent state.

SelfMem does not require that conclusion, and security research argues against it.

The write path is where memory becomes infrastructure

The systems characterization paper classifies memory into four broad paradigms:

1. long context;
2. flat sparse or dense retrieval;
3. structure-augmented retrieval, either append-only or consolidating;
4. agentic control flow in which the model decides when and how to read and write.

It profiles ten systems across construction, retrieval, and generation. Its most important finding is that rich memory moves work out of the visible query and into construction.

The paper reports:

• more than a 47× spread in energy per correct answer across evaluated systems;
• construction times ranging from under a minute for deterministic indexing to hours for LLM-mediated systems on the tested workload;
• a five-order-of-magnitude spread in per-session write latency;
• up to roughly 9× difference in projected storage footprint at 1M tokens per user;
• two orders of magnitude in effective time-to-first-token under identical serving hardware;
• super-linear construction-token growth for some agentic systems as the memory store grows.

The exact values are workload- and implementation-dependent. The structural point generalizes: memory has a write path, a read path, a maintenance policy, and a consistency model.

A cheap query can hide an expensive ingest. An asynchronous ingest can hide latency while serving stale memory. A small construction model can save money until it falls below the capability floor required to emit valid structured writes—at which point it can corrupt the store.

The paper’s recommendations translate into four production questions:

Choose the simplest memory paradigm that satisfies the task. Complexity is not intelligence; it is recurring construction, storage, and operational cost.

Better memory can create a better attacker

The memory-poisoning study examines four write channels and six attack classes across two agents, OpenClaw and HERMES, using MPBench’s 3,240 cases.

Under its controlled setup, the paper reports:

The authors attribute much of the gap to design. HERMES writes and carries memory forward more aggressively; OpenClaw has a more conservative retention policy and retrieves memory explicitly.

The attacks are not all obvious commands. Weak-signal cases look like plausible facts, policies, precedents, or successful procedures. That is why prompt-injection filters underperform: at the input boundary, a false precedent can look like legitimate content.

The paper’s strongest off-the-shelf detector reached 67.67% true-positive rate with 1% false positives under its protocol. Even that model-backed guard left meaningful coverage gaps, especially for weak-signal attacks. The study is limited to one underlying model and emulates some tool-delivered content as labeled context, so the rates should not be treated as production prevalence.

The architectural lesson is stronger than the benchmark rate:

Memory poisoning must be controlled on the write path and recall path, not only filtered on the input path.

If a memory system learns that “successful experiences” deserve reuse, an attacker needs only to make one malicious experience look successful.

Use three stores, not one magical memory

A governed design separates three jobs.

1. Evidence ledger

Keep the original interaction, tool result, document snapshot, and correction as immutable, source-addressable evidence. Protect it with tenant, identity, classification, consent, and retention controls.

The model can inspect it through scoped, read-only interfaces. It cannot rewrite history.

2. Candidate memory

Let the agent propose summaries, preferences, facts, procedures, and retrieval hints. Every candidate carries:

• source references;
• memory type;
• subject and tenant scope;
• classification and consent basis;
• valid-from, valid-to, and observed-at times;
• confidence and rationale;
• contradiction links;
• proposed retention and recall filters;
• proposing model and harness versions.

Candidate memory can be broad because it has no automatic authority.

3. Promoted memory

Only validated candidates become eligible for future compilation. Promotion can be deterministic, reviewed by a human, or both, depending on risk.

A promoted record is still filtered at recall by:

• tenant and subject;
• actor role and claims;
• task intent;
• data classification;
• consent purpose;
• freshness;
• contradiction state;
• policy and pack version.

This architecture preserves SelfMem’s task-sensitive workspace while preventing a model’s write preference from becoming the trust boundary.

A memory write is a decision

Treat every durable write as a typed decision, even if the storage engine calls it an upsert.

An illustrative, non-normative envelope:

memory_proposal:
  proposal_id: memp_042
  type: procedural
  content: "For vendor renewals, verify the signed price-cap clause."
  source_refs:
    - evidence_contract_884
    - correction_operator_117
  scope:
    tenant_id: tenant_acme_prod
    intents: [renewal.prepare]
  validity:
    observed_at: "2026-07-24T03:30:00Z"
    expires_at: "2027-07-24T03:30:00Z"
  controls:
    classification: CONFIDENTIAL
    promotion: reviewer_required
    recall_requires: [contract_evidence_access]
  status: candidate

The important fields are not the prose. They are the source, scope, validity, and promotion state.

For high-impact procedural memory, require replay: demonstrate which historical failures the proposed procedure fixes, and test adversarially whether a poisoned example can trigger the same promotion path.

Operate memory with lifecycle SLOs

Recall accuracy is necessary and insufficient. A production scorecard should include:

Do not co-locate unlimited background construction with latency-sensitive inference and hope the scheduler is fair. Apply admission control, batch deterministic embedding work, set iteration limits on model-controlled construction, and use freshness fences when a task cannot tolerate lag.

For example, a read-only personalization request may tolerate a memory snapshot several minutes old. A financial decision that depends on a just-approved limit should either wait for the write to commit or read the signed decision directly from the evidence ledger.

Consistency is an intent-level requirement, not one global setting.

The best-practice contract

The latest research supports an agent-controlled memory strategy—inside a system-controlled memory lifecycle:

1. Keep raw evidence immutable and addressable.
2. Let models propose memory; do not let proposals become authority automatically.
3. Require source refs, scope, validity, consent, and contradiction state on every durable candidate.
4. Separate captured, candidate, and promoted state.
5. Filter at both write and recall.
6. Choose memory complexity from workload capability, arrival rate, and query volume.
7. Measure construction, not only query serving.
8. Bound model-controlled loops with budgets, timeouts, and schema checks.
9. Evaluate poisoning alongside recall.
10. Make deletion, tombstoning, rollback, and replay part of the original design.

SelfMem is a meaningful result because it shows that the best memory strategy need not be a fixed, hand-written pipeline. The systems paper is a meaningful correction because it prices the whole lifecycle. The poisoning paper is a necessary warning because adaptable memory expands the write surface.

Combine all three:

Let the agent learn how to organize memory. Keep the right to trust, retain, and recall that memory outside the agent.

Research base

• SelfMem: Self-Optimizing Memory for AI Agents, submitted July 4, 2026.
• Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads, submitted June 4, 2026.
• From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents, MPBench and memory write-path analysis.
• A Survey on Long-Term Memory Security in LLM Agents, for the write–store–retrieve–execute–share–forget lifecycle.
• ContextOS: AI Agent Memory Is Broken, Promotion-Aware Memory, and Memory Fabric.

What to read next

• AI Agent Memory Is Broken
• Promotion-Aware Memory in Code
• The Identity Layer
• Memory Fabric