Self-Improving Harnesses: HarnessDev, HoH, and the Promotion Boundary

An agent edits its instructions, runs a benchmark, and reports a better score. Another agent works on a software project for several days and produces an impressive demo. Both may represent useful progress. They are different claims, and they need different evidence.

The September research wave makes it possible to separate improving a reusable execution system from improving the artifact produced inside that system. A practitioner should preserve that separation in architecture, evaluation, and release authority.

This analysis covers primary sources available through September 10, 2026. Research results below are attributed to their authors. The proposed operating design is an engineering recommendation, not a new ContextOS contract, an implemented runtime in this repository, or an independently reproduced performance claim.

Four research questions, four experiments

A system that performs well on the third question has not thereby solved the first. Its project can improve while its underlying harness remains unchanged. Conversely, a useful harness change can generalize across tasks without producing a spectacular single demo.

What HarnessDev contributes

HarnessDev separates a creator from the executor that runs the resulting harness. Creation covers six creator models, four domains, and 2,207 downstream instances. Evolution uses visible feedback and later evaluates frozen versions on disjoint SWE-Pro tasks. The authors report uneven creation quality, unstable improvements, and executor-dependent transfer. Human reference results are public system results rather than fully paired controls. Evolution has one trajectory per creator–runtime cell, with an unfinished cell; its limitations do not support population-level uncertainty claims. (HarnessDev, September 1, sections 3–6.)

The practical implication is to store two identities whenever a model proposes a harness change: the proposer and the intended executor. A change made by a strong coding model may be intended for a cheaper serving model. That is a deployment configuration to test, not an assumption that a model can use any instructions another model writes.

Store the parent candidate as well. A sequence of individually plausible edits can erase an earlier gain. Comparing only the final candidate with the initial baseline hides when regressions entered and which change caused them.

Search quality and execution quality are separate

HarnessOpt-Bench studies an optimizer editing a seed under an evaluation budget, with a trusted environment preserving versions and separating test access. Its 111 scored runs compare five optimizer models across four downstream tasks. The authors report that native coding harnesses are not consistently superior and that gains depend on the target task and seed. (Ursekar and colleagues, August 6.)

An earlier study, Harness Updating Is Not Harness Benefit, likewise separates producing useful updates from successfully using them. It reports different relationships between base model capability and those two abilities in its experiments. That is evidence against choosing an optimizer solely from a general task-solving ranking. It is not a universal prescription to use the cheapest model. (Lin and colleagues, May 28.)

The correct budget has at least two parts. Development cost includes candidate generation, smoke tests, and all selection trials. Deployment cost includes execution of the chosen candidate on future work. A large one-time search expense may be sensible for a high-volume workflow; it may never pay back for a rare task.

An illustrative break-even calculation is:

break-even accepted outcomes = total optimization cost / saving per accepted outcome

This requires a positive saving measured at comparable quality. Include human review, infrastructure, and maintenance if they change. Do not assume that a lower token count is automatically a lower business cost.

Why component boundaries matter

HarnessCompass proposes task-agnostic constraints, proactive feedback, and separate optimization of components before consolidation. Its reported SWE-bench Verified result moves from 54% to 66% under its GPT-5.4 setting after five iterations. The transfer and ablation claims belong to that experimental setup. (Zhang and colleagues, August 3.)

For implementation, the useful question is whether the patch has a comprehensible causal target. “Improve everything” can simultaneously alter retrieval, tool descriptions, compaction, and stopping behavior. If the score rises, the result is hard to explain; if it later falls, rollback becomes coarse.

Prefer a proposal such as: “Large test logs displace the unresolved task constraints; preserve those constraints while summarizing the logs.” Define the expected observation and a counterexample. Run the candidate against log-heavy tasks and against tasks where exact output details are essential. This yields a falsifiable change rather than a more elaborate prompt.

Agent feedback can help diagnose friction, but it is another observation. An agent saying a tool is confusing is not proof that the tool contract is wrong. Check the trace, the actual arguments, and the resulting state.

HoH is a different kind of progress

HoH organizes existing coding systems into planning, implementation, and testing iterations. Its GameCraft-Bench comparison includes repeated vanilla continuation. At three passes, the reported mean score is 71.52 for HoH and 58.24 for continuation, with 8.41M and 6.33M mean cumulative coding-harness tokens respectively. HoH at two passes also exceeds the three-pass continuation score while using fewer such tokens. Equal passes are not equal compute; the paper provides evidence beyond a single longer run, but not a complete operating-cost comparison. (Yan and colleagues, September 1, Table 2.)

The project README identifies a planner, developer, and read-only QA role, and describes a versioned artifact moving across iterations. At the review date it still announces HoH-lite as a future release. Treat the paper and demos as public research evidence; do not plan around a downloadable HoH-lite runtime without rechecking availability. (Project repository.)

The actionable pattern is to make each iteration produce a checkable state transition. A planner consumes requirements and prior evidence. A developer changes a bounded artifact. An evaluator records what it actually observed. The next iteration receives the artifact and evidence separately.

A practical separation of authority

The following is a proposed operating design. It can be implemented with ordinary processes, isolated workspaces, durable storage, and an existing CI system. Multiple conceptual roles do not require multiple concurrently running agents.

Enforce the separation through permissions, not by asking the proposer to be honest. If the same writable directory holds candidate code and authoritative test answers, a prompt saying “do not inspect the answers” is not a meaningful isolation boundary.

A release record should bind the candidate digest, executor identifier, environment, policy version, task-set revision, evaluation result, and parent candidate. These are example release-engineering fields, not replacements for RunContext, DecisionRecord, or any existing ContextOS type.

The corresponding flow is:

production failure evidence
  -> scoped proposal and candidate
  -> isolated development evaluation
  -> frozen selection
  -> protected quality, safety, and recovery evaluation
  -> authorized promotion or explicit rejection

The proposer can ask for evaluation. It cannot make a failed result disappear by submitting a different final message. Preserve unsuccessful candidates and incomplete trials with their actual status.

Durable progress requires more than a summary

For a long-running workflow, maintain a requirements ledger, an artifact revision, and an evidence bundle. A natural-language progress note is useful for orientation but should not be the only record of completion.

Suppose a coding agent reports that authentication is finished. Retain the requirement identifier, the artifact revision, the test or inspection evidence, and any unresolved limitations. After a context reset, the next worker can verify that claim against the artifact. If the artifact changed, the old evidence may need to be rerun.

Store failed checks too. Otherwise, the next iteration may mistake “never tested” for “working” or repeatedly rediscover the same defect. Make the distinction between verified, failed, and unverified visible in the continuation state.

This follows the existing ContextOS separation between decision evidence and promotion-aware memory. A repeated assertion is not automatically a durable fact, and a successful local experiment is not automatically a reusable policy.

Test interruption at the effect boundary

A multi-day coding demo mostly changes a workspace. A production agent may send a message, issue a refund, or modify access. Recovery must account for effects that occurred before a process failed.

Consider a timeout after an external service accepted a request but before the receipt was persisted. Repeating the request may duplicate the effect. Restoring a filesystem snapshot does not rewind the external service.

A useful fault test interrupts execution at that point, restarts the worker, and checks whether it reconciles the external state before acting again. Use a stable operation identity where the service supports it, preserve the uncertain status, and require postcondition evidence. If reconciliation is impossible, route the case to an explicit recovery path rather than manufacturing certainty.

The HarnessRisk research makes lifecycle safety a distinct evaluation object. The recommendation here is to extend a team’s own quality trials with setup, state, action, and recovery cases appropriate to its authority. A paper’s benchmark coverage cannot substitute for those deployment-specific controls.

Promotion gates should survive the optimizer

Freeze the acceptance policy before the final trial. Require complete evidence, valid task coverage, preserved authority boundaries, and a successful rollback exercise where rollback is meaningful. Improvement in task score should not silently compensate for a newly introduced prohibited effect.

For statistical quality gates, choose the minimum worthwhile effect and acceptable regression margin from the application. Predeclare the unit of analysis, repeated-trial design, and treatment of infrastructure failures. Avoid borrowing a p-value threshold as a complete business decision rule.

Retest after changing the executor, provider path, tool set, or durable-state interpretation. A candidate is validated for a configuration and workload, not blessed forever. Also test removal of old heuristics: a component that once prevented failure can later impose unnecessary context or execution cost.

These gates implement the intent of the ContextOS improvement loop: the system can search for changes while promotion remains governed and evidence-backed.

Research questions that remain open

The most useful next studies would repeat independent optimization trajectories, compare against equally funded simple search, and report all development cost. They would test transfer to new task families and multiple executors, then examine whether gains survive environmental change.

For long-running development, repeat the workflow across varied projects, inject interruption, measure regressions to previously verified functionality, and include reviewer effort. A polished artifact is an important outcome. Sustained, economical, recoverable operation is a larger claim.

Sources

All sources accessed September 10, 2026; the papers below are preprints.

1. HarnessDev authors, ByteDance Seed and collaborators. HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?. September 1, 2026, v1.
2. Varun Ursekar and colleagues. HarnessOpt-Bench. August 6, 2026, v1.
3. Minhua Lin and colleagues. Harness Updating Is Not Harness Benefit. May 28, 2026, v1.
4. Luan Zhang and colleagues. HarnessCompass. August 3, 2026, v1.
5. Haoyang Yan and colleagues. Harness-of-Harness. September 1, 2026, v1; public project repository.
6. HarnessRisk authors. HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety. August 18, 2026.

What to read next

• The September research map.
• Benchmark costs and uncertainty in the public data.
• Experiments and future articles on the publishing roadmap.