A useful trace explains why a particular result deserves attention. It can still be a poor regression test. If a coding agent examines the failure, writes the fixture and grader, and then changes the harness until that grader passes, it may fix the wrong thing. The workflow needs a way to turn observations into hypotheses without letting them set the rules for release.
NVIDIA’s October 21 Agent Toolkit expansion names NeMo Compass among its agent development tools. A November 4 engineering walkthrough shows a more interesting piece: the path from recorded failures through NeMo Eval Author to a measured harness change. This is distinct from the execution boundary in our OpenShell and Sentry assessment. Here the question is whether an engineering team has enough evidence to accept a proposed fix.
What the walkthrough actually establishes
NeMo Compass is a public research preview that analyzes traces for patterns such as anomalous behavior, recurring tool problems and mismatches with stated agent requirements. It gives supporting trace references that a person can inspect. NeMo Eval Author is a separate research preview that helps a coding assistant create or assess evaluations in Harbor or NeMo Gym. Neither project’s preview status establishes production suitability.
The November 4 walkthrough uses a fictional workplace and 42 recorded traces. The team turns an issue involving missing confirmation into four tests. On two withheld tasks, three runs per task, NVIDIA reports four passes out of six before the harness change and six out of six afterward. These are six runs over two held-out tasks, not six independent tasks or a measured production failure rate.
The same walkthrough describes an engineering example with Dassault Systèmes that reported malformed tool calls falling from 142 to zero after a targeted change. It separately describes an overnight optimization pass reporting a 28% relative improvement in an overall workflow score and 35% fewer tool calls per run across six checked scenarios. Those are reported observations in a specific environment, not an estimate of the marginal return from installing Compass. Engineering example
Keep the trace and the explanation separate
An observation might be that a support agent requested the same document four times. The candidate explanations differ: perhaps a tool returned an unexpected shape, a router restarted state, or the model ignored a perfectly usable result. If the trace export missed the tool result, the observer may not be able to tell. Instrument the gap before authoring a test that assumes a cause.
The Compass file schema guide distinguishes trace IDs, session IDs and task case IDs, and records limitations of trace conversion. That distinction matters for evaluation design. Ten retries on the same request can give ten traces but one underlying case. Treating them as independent yields an inflated estimate of how common the problem is and can put almost identical work on both sides of a split.
For a candidate finding, record the observed discrepancy, the affected version, the strongest competing explanation and one experiment that would distinguish them. Preserve an unmodified authorized trace reference. A summary is useful for routing the investigation, but should not become the sole record of what the target agent did. Trace contents may contain sensitive data or instructions from a third party; approve the analyst’s data route and restrict its access to the evidence needed for review.
Design the grader to catch both overaction and inaction
Suppose an agent preparing an internal note should send it only when a separate application condition is met. A grader that merely checks “no send” rewards a harness that disables all sends. A test for the useful successful path should show that a valid request produces exactly one completed transmission with the expected properties. A deny case should show that the note stays a draft when permission is missing. A result that only checks the wording of the final message cannot distinguish an actual transmission from a confident claim.
This is a proposed application-owned test. Use a deterministic in-memory or test-owned service to distinguish attempted, confirmed and rejected operations. Have someone other than the patch author challenge the grader with a candidate that does nothing, one that returns a plausible explanation despite a missing effect, and one that performs the right operation twice. These are tests of the oracle, not additional ordinary tasks to optimize against.
Keep enforcement outside the instruction change. An improved prompt can reduce undesirable attempts; it does not show that an outside service would block the next invalid request. Use current authority and external outcome checks from the actual application wherever the action matters. The trace study and the application control answer different questions.
Keep the final cases out of the loop
Split by case family, not by trace row. Put the traces and nearby reproductions for an observed failure in development. Use a second set for choosing among plausible fixes. Seal the final cases and their graders until a release candidate is ready, and record who has accessed them. If a finding from the final set informs another patch, retire that set to development and replace it before making a held-out claim.
| Proposed study arm | What it can establish |
|---|---|
| Existing harness on a frozen set | Baseline pass rates, false positives and resource use |
| Single symptom-targeted harness change | Whether the candidate improves the defined task |
| Independent variants and positive controls | Whether the fix still performs authorized work and transfers beyond the original trace |
| Missing or reordered trace spans | Whether the analyst flags missing evidence instead of assigning a confident cause |
Use both run-level and task-level reporting. Preserve wrong attempts, retries, aborted sessions and incomplete traces. A reduction in tool calls is good only if useful completion survives; a drop caused by early abandonment looks efficient in the wrong denominator.
Count the time from signal to accepted change
The proposed economic unit is an accepted, maintainable fix with a surviving regression test. Count the analysis calls, trace review, fixture design, verifier review, failed patch attempts, reruns and operator time. Compare that total with a conventional incident triage for similar cases, including cases where neither method identifies a defensible cause. Measure precision of findings and time to a decision alongside end-to-end task quality.
This makes a disappointing pilot informative. An analyst might find real issues but generate too many findings to review. A grader may be precise on known failures while missing ordinary successful work. Start in an offline environment with clearly labeled synthetic traces, then expand only if the process can explain why a proposed change improves the target workload without moving the acceptance criteria.