From Feature Stores to Agent Memory: What Production ML Teaches Agent Systems

An agent evaluated on yesterday’s support cases suddenly looks better after a memory backfill. The model has not changed. Neither has the retrieval algorithm. The evaluation is now retrieving a correction written this morning.

That is future information entering a historical decision. Production ML has spent years dealing with this family of problem. Agent memory should inherit that experience before inventing another vector-store abstraction.

The central lesson is that memory is a governed, time-dependent data product. Retrieval quality matters, but so do identity, availability, lineage, materialization, and the difference between what is known now and what was available then.

This article combines primary research and Feast documentation checked on September 6, 2026 with an original memory design proposal. The examples and schemas are illustrative; no production deployment or measured improvement is claimed.

What production ML already learned

Hidden Technical Debt in Machine Learning Systems identifies system-level risks including data dependencies, feedback loops, entanglement, and undeclared consumers. An agent memory store has analogous risks: one extraction change can affect every workflow recalling its outputs, while agents can create the very data that later guides them.

The ML Test Score turns readiness into concrete testing and monitoring needs. The transferable lesson is to evaluate the data and serving system around the learned component. High retrieval accuracy alone does not establish that memory is appropriate to use.

These are architectural analogies, not evidence that agent memory and ML features are identical. Features usually feed a predictor under a defined serving contract. Agent memory can contain natural language, disputed claims, private preferences, or malicious instructions. It can also be written by the system that later consumes it.

Map the useful concepts precisely

The mapping is a proposed design aid. It is not a claim that installing a feature store implements an agent memory policy.

Point-in-time correctness has two clocks

Feast’s point-in-time join documentation describes selecting historical feature values relative to each entity timestamp and a TTL window. Its current documentation also distinguishes event time from availability: by default, created timestamps deduplicate equal event timestamps without necessarily excluding later backfills. The documented filter_by_created_timestamp=True option adds an availability constraint where supported. Unsupported stores raise an error; the created timestamp must actually reflect availability for the intended guarantee.

For agent memory, we propose separating at least four times: when the fact applies, when the source was observed, when a candidate was recorded, and when it became eligible for recall. A user preference expressed on Monday but promoted on Wednesday was not available as promoted memory for Tuesday’s run.

Consider a synthetic sequence:

A Tuesday replay must not retrieve the Wednesday promotion. A current overnight-flight task should consider Thursday’s scoped correction. A retrospective analysis asking “what do we now believe about Tuesday?” is a third query. Mixing these questions makes a memory system look more capable and an audit less faithful.

Event time answers when the fact applies. System history answers what the runtime could access. Policy and promotion history answer whether it was eligible to use.

Store history, serve a projection

The proposed design separates immutable historical versions, governed eligibility events, and replaceable serving indexes. “Immutable” here means prior versions are not silently rewritten; retention and deletion policies still govern whether their payloads may be retained.

capture -> typed candidate -> review and contradiction checks
  -> promotion event -> authorized memory versions
  -> serving projection -> recall filters -> ranking -> context
 
correction / revocation -> eligibility change + projection invalidation

That aligns with the existing ContextOS memory fabric: only promoted memory reaches the compiler’s recall path. The following conceptual fields belong to an application storage design, not a replacement for MemoryProposal or another canonical type:

identity: tenant, subject, predicate, scope
value: typed payload or protected payload reference
lineage: source references, extractor version, parent memory versions
validity: valid_from, valid_to
history: recorded_at, superseded_at
eligibility: promotion event, revocation events, policy version
serving: projection revision, indexed_at, embedding model revision

Use intervals with a documented convention, such as inclusive start and exclusive end. Represent open-ended intervals explicitly. Do not overwrite a prior version’s value when correcting it; append a new version and preserve the transition metadata while retention permits.

For a historical run, eligibility means more than valid_from <= run_time. It includes whether the version existed, was promoted, remained permitted, and was actually present in the serving view used by that run.

Eligibility precedes similarity

The proposed recall order is:

1. Resolve tenant and canonical entity identity.
2. Apply the caller’s role, classification, consent, and purpose restrictions.
3. Select the relevant validity and system-history intervals.
4. Require promotion and exclude revoked or superseded versions for the selected query mode.
5. Apply freshness and conflict policy.
6. Rank eligible candidates and assemble bounded context.

Similarity can rank candidates inside that set. It cannot make an otherwise forbidden record eligible. If the storage engine retrieves a broad candidate set internally, filtering must occur before any text or metadata reaches the model, and the storage boundary itself must be authorized for that access.

Conflicting memories need explicit handling. “Prefer aisle” and “prefer window on overnight flights” can be a general rule and a scoped exception. “Refund already issued” and “refund not issued” for the same transaction may be a temporal update or an unresolved source conflict. A single confidence score cannot determine which interpretation is valid.

Use the existing memory model’s deterministic resolution and correction rules. When a required fact remains unresolved, expose the conflict through the established decision path rather than selecting the nearest embedding.

Separate logical history from actual serving history

A memory can be promoted at 10:00 but indexed at 10:04. A replay at 10:02 must decide what it is trying to reproduce.

Logical eligibility replay asks what the policy would have admitted from the authoritative store. Observed serving replay asks what the production recall path actually returned. Both are useful; they are not interchangeable.

Record the memory version IDs returned by recall, eligibility-policy version, projection revision, retrieval settings, and resulting context artifact. Preserve protected payloads only under the applicable retention policy. If they are no longer available, mark the replay partial rather than regenerating them from today’s store.

For approximate vector search, pinning an embedding model does not guarantee the same candidate order after an index rebuild. Persist the observed recall result when exact reconstruction matters. Use reruns against a rebuilt index for evaluation, clearly identified as new execution.

This distinction also exposes a subtle serving bug: an authoritative revocation may precede index deletion. The recall boundary needs a current eligibility check or another enforced invalidation mechanism; a stale vector index cannot remain the permission authority.

Treat memory promotion as a release

An extractor change can alter thousands of records without changing their source documents. A summarizer may drop negation, widen a preference’s scope, or turn a tentative statement into a durable fact.

Version extraction, normalization, embedding, and promotion policy separately. Before promoting a new extractor, compare it against held-out captures with typed expected outcomes: correct entity, predicate, scope, temporal interval, evidence reference, and eligibility decision. Review contradictions and unsupported assertions as first-class errors.

Use shadow extraction to compare candidates without making them recallable. A canary can promote selected low-risk classes under policy, with a defined rollback or quarantine operation. Rolling back a model alone does not withdraw memories it already wrote; lineage must identify the descendants to quarantine.

The W3C PROV data model provides a useful vocabulary for entities, activities, agents, and derivations. For memory, the practical requirement is to trace a served assertion through its extraction and promotion activity to attributable evidence. A timestamped transcript by itself does not supply that dependency graph.

Prevent the self-confirming memory loop

A model guesses that a customer has premium status. A summary stores the guess. Another run recalls it and offers a premium concession. The successful tool response is then summarized as evidence of premium status.

This synthetic sequence has produced apparent corroboration without an independent source.

Distinguish user assertions, authoritative observations, model inferences, and verified outcomes. Repeated derived claims should retain the same ancestry rather than count as independent support. A tool’s successful execution proves that an operation occurred under its semantics, not every premise used to justify it.

CaMeL’s work on untrusted data and capabilities reinforces a separate boundary: remembered text must not acquire authority just because it persists. The proposed memory pipeline preserves evidence and permissions; it never promotes a sentence such as “ignore approval checks” into a policy instruction.

Correction and deletion are distributed operations

A correction must reach the authoritative record, serving projection, caches, summaries, and any other derived stores that can reintroduce the old value. A deletion policy needs the same dependency inventory, including evaluation datasets and backups under their retention schedules.

This is an engineering propagation requirement, not a claim of legal compliance. Use a tombstone or equivalent eligibility event to block recall while asynchronous removal proceeds. Keep any retained audit metadata within the governing policy, and avoid treating a linkable hash of sensitive content as automatically anonymous.

Current revocation takes precedence over reproducing old private content for an unauthorized caller. Historical eligibility does not grant present-day access. If deletion prevents exact replay, record that limitation. An honest partial audit is preferable to resurrecting data from a forgotten cache.

A test suite that measures memory as a system

Measure eligibility violations, stale-recall rate, correction propagation time, recall precision among eligible records, contradiction handling, downstream trusted completion, and memory-caused regressions. Compare no-memory, current-memory, and historical-memory conditions on the same held-out tasks. Include the costs and errors introduced by writes, not just the benefit of reads.

No such experiment is reported here. The proposed test suite is an implementation target. Production ML’s contribution is the discipline to make memory a reproducible data dependency with an owner, a history, a serving contract, and an invalidation path.

What to read next

• Memory as a systems workload develops the storage and operations perspective.
• Memory Fabric describes the existing promotion and recall contract.
• Agent Harness Benchmark protocol shows how to evaluate memory alongside the rest of the execution system.