Operating Agent Products: Feedback, Rollout, and the Improvement Loop

Shipping an agent is not the finish line.

It is the point where the product starts producing evidence.

Every run creates traces. Every correction reveals a gap. Every approval delay shows operational friction. Every tool denial says something about authority or context. Every customer complaint may be a policy, UI, context, or workflow bug.

The product manager’s job after launch is to turn those signals into governed improvement.

In ContextOS, that is the Improvement Loop: observe, capture corrections, synthesize insights, compile strategy rules, queue research, propose tuning candidates, gate, release, and roll back when needed.

October 16 contract revision: proposal/autotune YAML and operating cadences below are illustrative application profiles, not exported runtime schemas, executed improvement results or scheduled automations. PM priorities cannot lower mandatory controls or replace current independent release authority.

The PM operating model

A real agentic product needs three cadences:

This is not bureaucracy. It is how product learning becomes runtime improvement instead of folklore.

What changes after launch

Before launch, you ask:

Can this work?

After launch, you ask:

What did real work teach us about the harness?

The data changes:

Each signal should have an owner and a route.

Route feedback into typed buckets

Do not let every failure become “improve the prompt.”

Use this routing table:

This table keeps the team from treating the model as the only change surface.

The weekly scorecard review

Run this every week during rollout.

Agenda:

1. What changed in the harness tuple?
2. Did Policy or Safety floors hold?
3. Which intents improved Utility?
4. What happened to latency and economics?
5. Which traces represent new failure modes?
6. Which operator corrections repeated?
7. Which approvals are creating bottlenecks?
8. Which proposals should be approved, rejected, or sent back?
9. Should rollout advance, pause, or roll back?

Decision options:

The review should end with a decision, not a discussion.

The proposal queue

Every improvement should be a proposal with eligible evidence. Validate correction origin/current identity, scoped access, deduplication and provenance before storage. A customer comment or repeated correction can inform a candidate but cannot grant production policy authority.

proposal:
  id: prop_vendor_onboarding_2026w20_07
  source:
    - correction: corr_182
    - trace: trace_vendor_771
    - decision_record: dr_vendor_771
  target:
    intent: vendor.onboarding.compliance_check
    layer: planner
  change:
    add_step: bank_verification_before_sanctions_summary
  expected_effect:
    utility.operator_correction_rate: -0.04
  guardrails:
    policy_score: 1.0
    safety_score: 1.0
    latency_p95_delta: "<= +10%"
  evals:
    search_set: vendor_compliance_search_v3
    release_set: vendor_compliance_release_v2
  rollout:
    start_stage: 0%_shadow
    rollback_tuple: vendor_harness_4.1.0

This is how a PM turns “the agent keeps missing bank verification” into a release-gated change.

Autotune with boundaries

Autotune is useful only when the PM defines the search space.

Bad:

Improve onboarding agent quality.

Good:

autotune_request:
  intent: vendor.onboarding.compliance_check
  primary_metric: operator_correction_rate
  direction: decrease
  allowed_surfaces:
    - context_pack.evidence_order
    - planner.tool_preference
    - response_examples
  forbidden_surfaces:
    - approval_modes
    - policy_rules
    - destructive_tool_access
  guardrails:
    policy_score: 1.0
    safety_score: 1.0
    p95_latency_delta: "<= +10%"

The product, policy and data owners define proposed surfaces within their current authority. Mandatory profile controls remain fixed; the optimizer proposes and an independent current release authority approves the full exact rollout action/evidence/immutable pins with required entitled quorum. A score or approver-name string is not permission. Freeze before sealed evaluation and bound repeated submissions/feedback.

Treat rollout as operations

A staged rollout is not a launch ritual. It is a learning plan.

Each stage should have:

• entry criteria,
• exit criteria,
• rollback trigger,
• support owner,
• scorecard window,
• open-question list.

Current owned tenant/audience/purpose and all ancestor restrictions/revocation constrain operators, scheduled optimizers, children and reviewers. Actual active containment/credential/data routes and provider preconditions remain independent; policy-expression errors stop compilation with structured diagnostics. Protected atomic stores and generation fencing govern wake/resume with remaining original shared spent/reserved capacity. Unknown provider effects retain stable identity/reservations until reconciled; keys, cancellation and tuple rollback cannot prove stop or reversal. Shadow requires actual enforced effect/data boundaries.

Govern collection and all nine data stages, actual readers/recipients/processors/retention and derived correction/trace/candidate copies with propagation receipts. Minimal eligible audit metadata does not justify permanent raw transcripts. Exact profiles and independent current conformance need useful, denial and fault evidence for the actual workflow/release/environment, beyond source tests or scorecards.

Incident review for agent products

When an agent incident happens, avoid blame-first reviews.

Use the trace:

Proposal disposition is one incident-review output, not proof containment or repair completed. Closure also needs actual effect reconciliation, verified control restoration and honest outstanding copy/usage state.

PM metrics after launch

Track these by intent and version:

Do not average all intents together. A safe average can hide a dangerous high-risk path.

Refresh the evals

Production changes the distribution.

Every month:

• add eligible reviewed correction cases to dev,
• add recurring failures to search,
• curate a separately protected release set; do not reuse tuning cases or leak its repeated feedback,
• retire stale examples,
• rebalance rare high-risk cases,
• review grader drift.

The eval set is not static documentation. It is the product’s regression memory.

What the PM should not delegate

PMs can delegate implementation. They should not delegate these decisions:

• Which intent matters most?
• Which error is worse?
• Which human owns approval?
• Which score is a hard floor?
• Which rollout stage is acceptable?
• Which feedback becomes a release candidate?
• Which incidents justify pausing automation?

These are product decisions wearing technical clothing.

Operating checklist

Every shipped agentic product should have:

• an owner by intent,
• a weekly scorecard,
• a trace review queue,
• a correction capture flow,
• a proposal queue,
• release gates,
• rollback tuple,
• support runbook,
• monthly eval refresh,
• incident review template.

If those do not exist, the product is not operated. It is merely deployed.

What to read next

• Product Management series
• Improvement Loop
• Evaluation and Observability
• Pack rollout in five stages
• From correction to StrategyRule
• Autotune the harness improvement loop