Agent Memory Should Be Compiled at Recall, Not Replayed From Storage

Retrieval answers only the first question.

It asks:

Which past experience looks similar to this task?

An acting agent must answer a sequence of stricter questions:

Relevant?
  -> Applicable?
  -> Trustworthy?
  -> Permitted?
  -> Current?
  -> Useful enough?
  -> Compile

A memory can be semantically relevant and operationally wrong. “Buy when all constraints match” is useful shopping guidance—until the current price is above the user’s limit. “Search the countertops after the drawers are empty” is a good household strategy—until the target is visible in the open drawer.

MemHarness, submitted July 30, puts an explicit reconstruction step between retrieval and action. A policy compares each retrieved experience’s source observation with the current state, rewrites the transferable part into current guidance, or emits <EMPTY> and acts without memory.

The paper reports strong gains on ALFWorld and WebShop, including out-of-distribution settings. Its most important contribution is architectural:

Retrieval relevance is not action applicability. Memory needs a state-conditioned compilation step before it can guide a decision.

Memory should not be replayed as a static prompt fragment. It should be compiled for the present.

For production, that step must remain a derived, ephemeral artifact. It may adapt guidance. It must not overwrite the historical record, erase provenance, widen authority, or silently promote reconstructed text back into durable memory.

This produces three progressively stronger claims:

1. Retrieval is not recall. Vector search finds candidates.
2. Recall is reconstruction. Candidates must be compared with the state that made them useful.
3. Production recall is compilation. The runtime combines memory with current state, evidence, policy, authority, intent, and budget before anything reaches the agent.

This post reviews a July 2026 preprint. Its benchmark results are not production proof, and the governance architecture below is our operating inference rather than a claim made by the paper.

What MemHarness actually does

Each memory entry contains:

• an abstract natural-language experience;
• the historical observation from which it was derived.

At decision time, the agent:

1. observes the current task and a recent history window;
2. decides whether to query memory;
3. retrieves the top three semantically relevant entries;
4. critiques each experience against its source state and the current state;
5. retains, revises, or rejects the experience;
6. generates an action from the reconstructed guidance.

If no memory applies, the reconstruction returns <EMPTY> and the agent falls back to self-reasoning.

The paper describes this as a five-stage conceptual process—observation, retrieval, critique, reconstruction, and action—and a three-stage inference pipeline in which critique is folded into retrieval, reconstruction, and action generation.

The same Qwen2.5-7B policy performs reconstruction and action generation. It is first aligned to the interaction format with a small cold-start dataset, then trained end-to-end with Group Relative Policy Optimization. The primary reward is task success, with a small format bonus.

This gives the system a useful middle ground:

explicit memory
  -> inspectable historical record
  -> state-conditioned reconstruction
  -> adapted guidance
  -> action

Pure parametric learning adapts behavior but hides which experience influenced a decision. Static retrieval preserves an explicit record but injects it without adaptation. MemHarness keeps the record visible while learning how to use it.

The results show negative transfer, not just better recall

The main comparison:

Raw memory reduced ALFWorld performance below the memory-free policy. Retrieval added relevant-looking information and made the agent worse because the historical strategy did not fit the current state. WebShop behaves differently: raw memory improved on GRPO there, and reconstruction improved it further. Memory replay is not universally harmful; its risk is negative transfer under state mismatch.

The inference-time ablations reinforce the point. These variants use the MemHarness-trained policy and disable components only during evaluation; they are not the same experiment as the separately trained raw-memory baseline above.

The no-memory result is consistent with reconstruction training improving the underlying policy, although this ablation does not isolate reconstruction from every other difference in the MemHarness training procedure. A generic instruction-tuned rewriter did not reproduce the full result; adaptation needed to be coupled to task outcomes.

On ALFWorld’s unseen layouts and object configurations:

The result supports a precise claim:

When state varies, directly replaying retrieved experience can cause negative transfer, and an explicitly trained applicability step can reduce it.

It does not establish that every production memory should be rewritten by the acting model.

Source state is part of the memory

MemHarness stores the historical observation beside the abstract experience. That choice is load-bearing.

Removing source state reduced success from 85.2% to 80.0% on ALFWorld and from 75.6% to 73.0% on WebShop. On ALFWorld, replacing it with a random historical state still produced 84.3% success, but the per-memory rejection rate rose from 8.7% to 13.3%. On WebShop, rejection rose from 56.0% to 63.3%. The model was using the contrast between “then” and “now,” not merely polishing retrieved text.

Production memory often stores only:

experience: "Refresh inventory before retrying after a stale-availability failure."
embedding: [...]

That is not enough to decide applicability. The record also needs:

• the event or evidence that produced it;
• the state in which it was observed;
• tenant, role, and data-classification scope;
• validity conditions and exceptions;
• policy and tool versions;
• observed time and freshness window;
• promotion decision and reviewer;
• contradiction and supersession links.

Without source context, reconstruction becomes confident improvisation over an orphaned sentence.

A memory without the state that made it useful is an orphaned conclusion.

Memory, state, evidence, policy, and authority are different things

Historical experience is not authoritative merely because it survived promotion. Production systems need an explicit authority taxonomy:

Type alone is not enough. Each item also needs an authority source: inferred, user-declared, system-observed, externally verified, or an authoritative control. In ContextOS, promoted memory and retrieved evidence remain data_only; only governed pack, policy, and tool-registry controls can carry instruction authority.

Reconstruction is a context compiler stage

The right production architecture keeps historical learning, current truth, and control authority separate:

HISTORICAL / LEARNED
promoted memory
      |
      v
deterministic eligibility
tenant + role + consent + classification + freshness
      |
      v
semantic retrieval -----------------------+
      |                                    |
      v                                    v
candidate memories + source state    CURRENT STATE + INTENT
      |                                    |
      +------------------+-----------------+
                         v
               typed reconstruction
                         |
                         v
CURRENT VERIFIED EVIDENCE -> ground and validate
POLICY + AUTHORITY ----> constrain and authorize
                         |
                         v
                  context compiler
              budget + priority + provenance
                         |
                         v
                       agent

Each stage answers a different question:

Collapsing these into vector similarity gives retrieval authority it does not deserve. Collapsing them into one model call gives the model policy authority it should not have.

Verified current state and evidence from their systems of record must be available to reconstruction because they determine applicability. They remain data_only, not control instructions. Policy and authority operate as hard controls both before memory visibility and after reconstruction; they are not merely extra prompt text.

Durable learning follows a separate path:

action outcome
  -> trace and evidence
  -> typed memory proposal
  -> deduplication + contradiction + consent + trust review
  -> promotion
  -> memory fabric

There is no direct write-back edge from reconstructed guidance to durable memory.

Applicability score is not retrieval score

Do not hide memory decisions inside one similarity number:

retrieval_score      = 0.94
applicability_score  = 0.21
provenance_trust     = 0.98
freshness_score      = 0.77
expected_utility     = 0.31

A memory can be highly relevant and well sourced but completely inapplicable. It can be relevant and applicable but expired. It can fit the state but fail consent or tenant scope.

Keep hard gates separate from learned scores:

Use(memory, state) =
  Eligible(memory, run_context)
  AND Relevant(memory, state)
  AND Applicable(memory, state)
  AND EvidenceValid(memory, state)
  AND ExpectedUtility(memory, state) > threshold

Eligible includes policy, consent, classification, tenant, and authority checks. Those checks should not be averaged into a composite score that can trade a permission failure for high semantic similarity.

Keep original memory and reconstructed guidance separate

The reconstructed output should never replace its source, and it should not primarily be an arbitrary natural-language instruction. Produce a typed artifact first; render model-facing text only after validation.

An illustrative envelope—not a new core ContextOS type—looks like this:

promoted_memory:
  memory_ref: mem:reservation_retry:044
  version: 3
  promoted_status: approved
  semantic_class: learned_heuristic
  authority_source: system_observed
  integrity: verified
  instruction_treatment: data_only
  content: "Refresh inventory before retrying after a stale-availability failure."
  source_refs:
    - trace://run-293/step-18
  source_state_ref: state://run-293/step-18
  source_state_hash: sha256:81c...
  evidence_refs:
    - tool://inventory.lookup/call-881
  valid_from: 2026-04-01T00:00:00Z
  valid_until: 2026-12-31T23:59:59Z
  tenant_scope: tenant_acme_prod
  classification: INTERNAL
 
reconstruction:
  reconstruction_id: rec_01J7MEM9
  run_id: run_a1b2c3d4e5f60718
  source_memory_refs:
    - mem:reservation_retry:044
  current_state_ref: state://run-a1b2/step-7
  current_state_hash: sha256:a21...
  applicability:
    verdict: PARTIAL
    confidence: 0.91
  transferable_claims:
    - claim_id: claim_12
      content: "Refresh current availability before considering a retry."
      evidence_refs:
        - tool://inventory.lookup/call-881
  rejected_claims:
    - claim_id: claim_13
      reason: SOURCE_INVENTORY_VERSION_MISMATCH
  predicates:
    matched:
      - previous_failure == STALE_AVAILABILITY
    mismatched:
      - source_inventory_version == current_inventory_version
  proposed_guidance:
    decision_key: reservation_retry
    outcome: refresh_evidence
    required_evidence:
      - inventory.current
    prohibited_effects:
      - purchase.commit
  provenance:
    reconstructor:
      model_ref: qwen/Qwen2.5-7B-Instruct@sha256:4f2...
      tokenizer_ref: qwen2.5@sha256:90a...
      prompt_template: memory-reconstruct@7
      runtime_ref: inference-runtime@3.2.1
      decoding:
        temperature: 0
        seed: 42
    retrieval:
      retriever_version: semantic-retriever@5
      embedding_model: BGE-M3@sha256:1d8...
      index_snapshot: memory-index://tenant-acme/2026-07-31T04:00Z
      scores:
        "mem:reservation_retry:044": 0.94
    context:
      state_ref: state://run-a1b2/step-7
      state_hash: sha256:a21...
      evidence_snapshot: evidence://tenant-acme/2026-07-31T04:12Z
      policy_bundle: policy://reservation-controls@12
    output_artifact_ref: artifact://run-a1b2/reconstruction-rec_01J7MEM9
  validation:
    policy: passed
    evidence: passed
    authority: passed
  expires_at: 2026-07-31T04:30:00Z

Only after validation should the Context Pack Compiler render approved claims and guidance into natural-language or structured model context.

The reconstruction is run-scoped and expires. Preserve the exact output artifact together with pinned model weights, tokenizer, prompt, retrieval snapshot, source state, evidence snapshot, policy bundle, decoding configuration, and runtime. Those inputs make the decision attributable and replayable for audit. A seed alone does not guarantee byte-identical regeneration across model runtimes, so the audit contract retains the produced artifact rather than promising that stochastic execution can always recreate it.

If the reconstruction reveals a reusable lesson, emit a new memory proposal linked to the original trace. That proposal returns through deduplication, contradiction checks, consent rules, review, and promotion. Success is evidence for a candidate, not permission to mutate the store.

Learned applicability is not a trust boundary

MemHarness optimizes reconstruction through task reward. That is appropriate for learning performance in a simulator. It is not sufficient for production governance.

A reward-trained reconstructor can:

• preserve a strategy that wins the benchmark but violates current policy;
• adapt poisoned memory into more persuasive guidance;
• remove a caveat that mattered for safety but not task reward;
• overfit to simulator affordances;
• infer an unsupported exception;
• reject memory selectively while still using it implicitly in hidden reasoning;
• turn a historical outcome into normative authority.

Place deterministic gates around reconstruction:

promoted memory
  -> tenant, consent, classification, and freshness eligibility
  -> typed reconstruction
  -> tool allow-list validation
  -> approval-mode floor
  -> evidence coverage
  -> contradiction check
  -> authority check
  -> context budget
  -> eligible for compiled context

The model decides applicability inside the permitted set. The control plane defines the permitted set.

Memory poisoning makes reconstruction more important—and insufficient

A reconstruction layer can reject a memory whose state does not match. It cannot prove that the memory was trustworthy in the first place.

Poisoned experience can be:

• perfectly matched to the current state;
• written to resemble a successful procedure;
• scoped broadly enough to survive retrieval filters;
• crafted so the “adapted” version preserves the malicious instruction;
• reinforced by repeated attacker-generated trajectories.

This is why promotion-aware memory must precede reconstruction:

• captures remain immutable evidence;
• candidates are invisible to ordinary recall;
• promotion checks provenance, consent, classification, and contradictions;
• only promoted records become recall-eligible;
• reconstruction adapts eligible records but never upgrades trust.

Applicability answers “does this fit?” Promotion answers “may this be trusted?” Neither can replace the other.

Use a Memory Decision Scorecard

Memory evaluation usually rewards recall accuracy or aggregate task success. A reconstruction layer needs a scorecard that separates selection, utility, trust, and economics.

The North Star should be a paired counterfactual, not memory recall rate:

Verified Memory Lift =
  mean(
    outcome_with_reconstruction[i]
    - outcome_without_memory[i]
  )

Compute it on a predeclared set of matched scenarios, report uncertainty, and publish negative-transfer rate beside it. Conditioning the headline metric only on memories the model accepted creates selection bias because acceptance is part of the system under evaluation. Accepted-only lift is useful as a diagnostic slice, not as the primary claim.

Create matched triples for evaluation:

current state
  + applicable memory
  + minimally edited inapplicable memory

Run all three conditions:

1. no memory;
2. raw replay;
3. reconstruction.

Then test out-of-distribution state, memory formed under superseded policy, contradictory evidence, poisoned memory, missing source state, and revoked consent. Aggregate success can hide a dangerous false-accept slice.

The operational design needs two fallbacks

MemHarness uses <EMPTY> to fall back to self-reasoning. Production systems need to distinguish:

Treating both as “use your own reasoning” invites unsupported action.

The reconstructor should also expose structured rejection reasons:

• source-state mismatch;
• expired validity window;
• unresolved contradiction;
• policy-version mismatch;
• missing evidence;
• scope or classification mismatch;
• suspected poisoning;
• low expected utility.

Some reasons can be decided deterministically before the model sees the memory. Do that early to save tokens and reduce attack surface.

The limitations that matter

MemHarness is compelling research, but its production boundary is substantial:

• evaluation covers two simulated interactive environments;
• the acting and reconstruction policy is one 7B model family;
• the cold-start protocol uses GPT-5.1-generated retrieval queries and memory summaries before reinforcement learning;
• the counterfactual mechanism probes use GPT-5.1-generated minimally edited states;
• the memory bank is populated from the agent’s own sampled trajectories;
• the main reward is sparse task success plus a small format bonus;
• security, memory poisoning, privacy, consent, and cross-tenant isolation are not evaluated;
• natural-language source observations are a partial representation of historical state;
• the reconstruction is trained as latent reasoning rather than independently supervised truth;
• comparison with closed models does not hold harness, training, or inference budget constant;
• scale to open-ended production environments remains future work.

The reconstruction behavior is learned through reinforcement learning, but its initial interaction protocol and memory-abstraction format are bootstrapped from strong-model-generated demonstrations. GPT-5.1 is not used during RL or the main task evaluation. Its separate use to create the offline counterfactual probes should be understood as part of the evidence for the mechanism, not part of the reported execution policy.

The result earns a new component in the memory architecture. It does not earn autonomous durable self-rewriting.

The best-practice contract

A production reconstruction layer should satisfy twelve rules:

1. Store source state and evidence with every reusable memory.
2. Keep policy and entitlement outside memory; evaluate them from current authoritative controls.
3. Filter deterministic eligibility before semantic retrieval.
4. Treat retrieval relevance, applicability, provenance, freshness, and utility as separate decisions.
5. Reconstruct into a typed, run-scoped artifact with its own identity.
6. Never overwrite or silently summarize away the source memory.
7. Validate policy, evidence, scope, authority, and approval floors before compilation.
8. Render model-facing guidance only from validated typed fields.
9. Distinguish not_applicable from insufficient_evidence.
10. Send reusable lessons back as proposals, never automatic durable writes.
11. Evaluate paired lift, negative transfer, false acceptance, poisoning, and OOD state.
12. Retain the exact reconstruction artifact and trace which sources influenced guidance and action.

But compilation is not revisionism. The historical record remains immutable; the adapted guidance remains attributable; policy remains outside the model; and durable learning still passes through promotion.

Stored memory is attributable evidence from the past. Recall is the governed act of compiling that evidence against the present.

That is how memory becomes useful under change without becoming ungoverned authority.

What to read next

• Agent Memory Is a Systems Workload
• AI Agent Memory Is Broken
• Promotion-Aware Memory in Code
• Memory Model