The AI Software Delivery Squad: From Ticket to Proof-Carrying Pull Request

The dangerous question in AI-assisted development is: How much code can the agent write?

The useful question is: What evidence must accompany a change before a human or repository policy can trust it?

That shift—from generated code to evidenced delivery—changes the product. A production coding agent is not an autocomplete box with shell access. It is a bounded software-delivery system that turns a ticket into a proposed change, proves what it checked, exposes what it could not prove, and stops at the correct authority boundary.

The best output is not merely a patch. It is a proof-carrying pull request: a small, reviewable change bundled with the issue interpretation, scope, test evidence, risk assessment, unresolved uncertainty, and a replayable record of how the work was produced.

This is the operating model behind the ContextOS Software Delivery Squad.

A coding agent is a delivery system, not a code generator

Real software work is a sequence of decisions under incomplete information:

1. What does the ticket actually require?
2. Which repository rules and architectural constraints apply?
3. What is the smallest safe write set?
4. Which checks distinguish “implemented” from “looks plausible”?
5. Who owns the affected surface?
6. Which side effects may the agent perform, and which require another authority?

A single model can participate in all six. It should not silently own all six.

The durable architecture separates responsibilities so that an implementation cannot approve itself and a fluent explanation cannot substitute for an external check.

These can be separate agents, deterministic stages, or one agent operating under distinct contracts. The important property is not the number of models. It is separation of concerns with explicit inputs, outputs, and authority.

Why the ticket is the first test

Coding failures often begin before code is written.

OpenAI’s work validating SWE-bench found that many benchmark issues were underspecified, paired with unfair tests, or difficult to run reliably. The broader lesson is operational: an agent cannot compensate for a missing product decision by reasoning harder. It can only guess more persuasively.

Before planning, the Requirements Agent should compile the issue into a task contract:

intent: delivery.implement
acceptance_criteria:
  - expired sessions return HTTP 401
  - the response preserves the existing error envelope
  - active-session behavior does not change
non_goals:
  - no token-format migration
  - no changes to refresh-token rotation
constraints:
  - preserve the public API schema
  - do not modify generated clients
open_questions: []
required_checks:
  - targeted session tests
  - API schema compatibility
  - typecheck

If a material field is unknown, the correct output is not a patch. It is a clarification request that names the missing decision and explains how the possible answers change the implementation.

A useful readiness rule is:

ready_to_plan =
  acceptance_criteria_are_testable
  AND affected_behavior_is_identifiable
  AND material_product_choices_are_resolved

This is not bureaucracy. It prevents the most expensive form of rework: a technically clean implementation of an imagined requirement.

Compile repository context; do not dump the repository

The Codebase Agent’s job is to build a focused context package, not to fill a context window.

Start with the repository’s own authority hierarchy:

1. scoped agent instructions such as AGENTS.md,
2. contribution and architecture docs,
3. package and build configuration,
4. ownership rules such as CODEOWNERS,
5. nearby implementation and tests,
6. generated or vendored-file warnings,
7. recent history only when it resolves intent or convention.

The resulting file map should distinguish four sets:

This distinction matters because “repository write access” is too coarse. A documentation fix, an authentication change, and a database migration may all involve a branch write, but they do not carry the same consequence.

ContextOS represents that consequence as an action-risk vector, not one universal ladder:

The key design rule is simple: authority does not flow downstream just because earlier steps succeeded. Permission to edit does not imply permission to push. Permission to push does not imply permission to open a PR. PR creation does not imply permission to merge or deploy.

Make the plan an executable contract

“Fix the bug and run tests” is not a plan. It is an aspiration.

A plan becomes governable when another component can compare execution against it:

{
  "goal": "Reject expired sessions without changing active-session behavior",
  "write_set": [
    "src/auth/session.ts",
    "tests/auth/session.test.ts"
  ],
  "protected_surfaces": ["authentication", "public_api"],
  "steps": [
    "add a failing boundary test",
    "implement expiry validation at the session boundary",
    "run targeted tests",
    "run API compatibility and type checks"
  ],
  "stop_conditions": [
    "required change expands into token rotation",
    "public schema must change",
    "targeted test environment cannot be reproduced"
  ],
  "rollback": "revert the isolated patch; no migration or data write"
}

The Critic can now check concrete properties before any edit:

• Does each acceptance criterion map to a step or check?
• Is the write set proportionate to the task?
• Are owners known for protected surfaces?
• Are the proposed tests capable of detecting the requested behavior?
• Is rollback credible for the planned effects?
• Does the plan stop when its assumptions stop being true?

Plans for migrations, authentication, authorization, billing, cryptography, public schemas, or production configuration should trigger a named review gate before implementation. The gate should inspect the issue contract, proposed write set, owners, validation strategy, and rollback—not a paragraph of model confidence.

Isolate effects and keep the patch small

The Implementation Agent should work in an ephemeral branch or worktree pinned to a known base commit. It receives narrow credentials, explicit budgets, and no production secrets.

During implementation, three invariants matter:

edited_files ⊆ approved_write_set
executed_tools ⊆ allowed_capabilities
consumed_budget ≤ reserved_run_budget

If the agent discovers that a required fix crosses the boundary, it should stop, revise the plan, and re-enter the relevant gate. Quiet scope expansion is not initiative; it is a governance failure.

Patch size is also a quality signal. Large generated diffs increase reviewer load, hide accidental changes, and make causal debugging harder. Prefer a sequence of independently reviewable patches over one broad “complete” rewrite. Generated files, lockfiles, snapshots, and formatting churn should be called out explicitly because they distort the visible size of the behavioral change.

Validation needs positive, negative, and regression evidence

A green test command is evidence, but it is not the whole argument.

The validation bundle should answer three different questions:

That is the useful idea behind the FAIL_TO_PASS and PASS_TO_PASS split in SWE-bench: prove that the defect was fixed and that previously working behavior still works. In a production repository, add the checks implied by the touched surface—type checks, lint, schema compatibility, security scanning, migration validation, or a build.

The Validation Agent should record exact commands, exit status, base and patch commit, environment identity, duration, and whether the result was fresh or cached. It must distinguish:

• product failure: the patch violates an acceptance criterion;
• regression: established behavior broke;
• environment failure: dependencies, services, fixtures, or permissions made the check inconclusive;
• flaky result: repeated runs disagree;
• missing coverage: no available check proves the claim.

An unavailable test environment must not become “tests passed.” An unrelated failing test must not be hidden. An allowed waiver should name the failure, owner, rationale, expiry, and compensating evidence.

The review agent is a skeptic, not a second implementer

The Review Agent should receive the issue contract, plan, diff, validation bundle, ownership map, and risk policy. Its job is to find reasons the proposed change is not ready.

Review in this order:

1. Intent: does the patch satisfy every acceptance criterion and respect every non-goal?
2. Scope: are all edits inside the approved write set and free of unexplained churn?
3. Correctness: are boundary conditions, error paths, concurrency, and state transitions sound?
4. Security: did the change alter trust boundaries, validation, secrets, dependencies, or data exposure?
5. Operations: are observability, compatibility, migration, failure, and rollback addressed?
6. Evidence: can each readiness claim be traced to a check, artifact, or explicit human judgment?

The output should be typed:

{
  "verdict": "revise",
  "findings": [
    {
      "severity": "blocking",
      "category": "regression",
      "claim": "active sessions near the expiry boundary can be rejected early",
      "evidence_ref": "diff:src/auth/session.ts#L84",
      "required_action": "add clock-skew handling and boundary tests"
    }
  ],
  "checks_verified": ["targeted_tests", "typecheck"],
  "checks_unverified": ["api_compatibility"]
}

Free-form prose can accompany the verdict, but prose should not replace it. The orchestrator needs a deterministic answer: accept, revise, replan, reject, or escalate.

The proof-carrying pull request

Only after review acceptance should the PR adapter be surfaced. The PR body is assembled from run artifacts rather than improvised at the end.

A compact template:

## Intent
- Fixes: ISSUE-142
- Acceptance: expired sessions return 401; active sessions unchanged
- Non-goals: token rotation and token format
 
## Change
- Added expiry validation at the session boundary
- Added boundary and regression tests
- Scope: 2 behavioral files; no schema or dependency changes
 
## Evidence
- `npm test -- session.test.ts` — passed
- `npm run typecheck` — passed
- API compatibility check — passed
 
## Risk and review
- Surface: authentication
- Code owner: @identity-platform
- Automated review: accepted
- Human review: required by branch protection
 
## Rollback
- Revert this PR; no data migration or irreversible side effect

GitHub’s protected-branch controls can require reviews, status checks, code-owner approval, conversation resolution, and an up-to-date branch before merge. Use those controls as the enforcement boundary. Do not recreate merge policy inside a prompt, and do not give the agent a bypass simply because it produced the evidence package.

The DecisionRecord is the delivery receipt

The run should close with a DecisionRecord that binds the proposal to its inputs and proof:

{
  "decision_key": "delivery.pr.open",
  "subject_ids": ["repo_payments", "issue_142", "commit_8af31c"],
  "outputs": {
    "pull_request_ref": "pr_381",
    "base_commit": "commit_92fd10",
    "head_commit": "commit_8af31c"
  },
  "evidence_refs": [
    "artifact_issue_contract_142",
    "artifact_plan_142_v2",
    "artifact_test_run_991",
    "artifact_review_verdict_773"
  ],
  "approvals": ["approval_plan_identity_team_44"],
  "controls_active": [
    "WRITE_SET_ENFORCED",
    "NO_MERGE_CAPABILITY",
    "GATE_PR_REVIEW"
  ],
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

This is not a transcript of hidden reasoning. It is the operational receipt: what was requested, which versions and commits were used, what changed, which evidence supports the proposal, who approved gated steps, what remained outside the agent’s authority, and how to replay the run.

Roll out by authority, not by demo quality

The safest rollout expands both task coverage and side-effect authority deliberately.

Do not graduate on task-success rate alone. Track a scorecard:

• acceptance-criteria pass rate,
• first-pass targeted and regression-test rate,
• reviewer acceptance and substantive rework rate,
• scope-expansion and protected-file block rate,
• security finding escape rate,
• flaky or inconclusive validation rate,
• rollback and hotfix rate for agent-authored changes,
• median human review time,
• cost and latency per accepted pull request,
• replay success against the pinned environment.

The denominator matters. “PRs opened” rewards activity. “Accepted changes with preserved controls and no downstream incident” rewards delivery.

Failure modes worth designing for

The operating principle

AI can compress the distance from issue to candidate change. It cannot eliminate the need to define intent, constrain authority, validate behavior, preserve ownership, and make failure recoverable.

The mature goal is not an agent that can do everything a software team does. It is a delivery system in which every automated step leaves behind enough evidence for the next boundary to make a better decision.

Build toward the proof-carrying pull request. Let humans and repository policy retain the final word on merge and release.

Research base

• ContextOS playbook: Software Delivery Squad.
• ContextOS contracts: API Contracts, Governance, Harness Engineering, and Evaluation & Observability.
• OpenAI: Introducing SWE-bench Verified for the effects of underspecified issues, unreliable environments, and the distinction between tests that prove a fix and tests that guard existing behavior.
• GitHub: About protected branches for enforceable review, status-check, code-owner, and merge controls.
• NIST Secure Software Development Framework and NIST DevSecOps practices for verifiable secure-development processes and human monitoring of AI-generated content.
• OpenAI: Introducing Codex for repository instructions, configured environments, reliable tests, and manual validation of agent-generated code.

What to read next

• How Great AI Engineers Build Agents
• AGENTS.md Done Right
• Replay Is the Real Audit Log
• Software Delivery Squad playbook