An agent can improve because its model improves, because its runtime improves, or because the two become better matched. Qwen-Planner-Agent makes that interaction the subject of a research program. The evaluation challenge is to establish which improvement transfers beyond the feedback used to create it.
Alibaba’s MAI team submitted the paper on September 24, 2026. This assessment uses the v1 preprint, checked on October 6. It is research evidence, not a claim of a generally available consumer feature or an independently replicated system. Paper record
The contribution is a development loop
The authors connect data construction, model training, and harness adaptation through execution feedback. Their environments include programmatic sandboxes, language-model simulations, and selected real-device sessions. They distinguish task-completion verification from admission of a trajectory into training. The paper also describes fixed-checkpoint harness ablations and identifies the co-evolution feedback set as a development set rather than an independent final test. Qwen-Planner-Agent v1
That last distinction is central. A set can be excluded from direct gradient training while still influencing a system through repeated evaluation and harness revision. The absence of direct training exposure does not make repeated development feedback equivalent to an untouched test.
Our assessment is that this is a useful framework for studying coupled improvement. Its value depends on preserving the difference between optimization evidence and generalization evidence.
Environment fidelity determines what success means
A programmatic sandbox can verify that a database reached the desired state. A real device can expose permission dialogs, asynchronous interfaces, and authentication failures. A language-model simulation can cheaply broaden the situations an agent encounters, but its description of success may itself be mistaken.
These are different measurement instruments. Combining their trajectories into a single dataset requires retaining the environment type and verifier provenance. Otherwise, a model may appear to improve by becoming better at satisfying a simulator’s expectations while remaining brittle on actual devices.
Consider a hypothetical task to move a meeting and notify attendees. A simulator might report both operations as successful in one response. A real calendar may update immediately while email delivery fails. A training record that labels the entire task successful teaches a different closure condition from one that requires both postconditions.
A proposed reproduction should therefore stratify results by environment. Report how many tasks depend on real external state, how resets work, and which failures indicate invalid instrumentation rather than bad agent behavior. A corrupted test environment should not be counted as a clean success or quietly removed after inspecting which system it favors.
Attribution needs a crossed experiment
To investigate model–harness co-evolution, evaluate four combinations on the same untouched task set:
| Model | Harness | Main question |
|---|---|---|
| Baseline | Baseline | What is the starting capability? |
| Updated | Baseline | What transfers through model training alone? |
| Baseline | Updated | What transfers through runtime changes alone? |
| Updated | Updated | What does the combined system achieve? |
On a chosen additive outcome scale, the difference between the combined gain and the two separate gains estimates an interaction. It does not prove a universal synergy: the value depends on the task distribution, metric, and budget. Repeat runs and uncertainty estimates are needed because agent outcomes vary.
The strongest practical evidence is that an updated harness helps more than the checkpoint it was tuned around, or that an updated model remains useful with a different but compatible harness. A gain confined to one exact pairing may still be commercially valuable, but it should be described as specialization.
Efficiency rewards can alter task strategy
The paper’s CARE method explicitly addresses reasoning and tool-use efficiency while preserving task performance. That is a worthwhile research objective, but the deployed meaning of efficiency needs independent scrutiny. CARE and training analysis
An agent can reduce calls by skipping verification. It can reduce reasoning by abandoning difficult tasks. It can compress an answer until important uncertainty disappears. A cost metric must therefore be paired with independently checked completion and the frequency of omitted prerequisites.
For the meeting example, one fewer call is an improvement if a redundant lookup disappears. It is a regression if the agent stops checking whether attendees were notified. The evaluator must distinguish these paths rather than rewarding a short trace on its own.
Memory should be tested against changing reality
Persistent memory creates another transfer problem: historical success does not guarantee present applicability. A remembered device setting, contact, or preference can become obsolete.
A proposed evaluation should introduce explicit corrections, conflicting live state, and requests to forget. Test whether retrieved memory remains a claim with a source and scope, or becomes an unquestioned instruction. Keep memory-update assessment separate from task success; a task can finish correctly while leaving a harmful persistent record.
The follow-up study I would prioritize freezes a final test set before the development loop begins, reserves unseen applications and temporal changes, and evaluates all four model–harness combinations under equal budgets. Its outputs should include failures, verifier decisions, environment type, and total optimization cost. That would turn a promising co-evolution result into more informative evidence about portability and sustained improvement.