Skill Lift: How to Gate Agent Skills With Paired Live Evals

A skill can be perfectly formatted, clearly written, free of obvious secrets, and useless at runtime.

The agent may never discover it. It may read the file and call the wrong script. It may execute the correct script with the wrong arguments. It may get the right result and misreport it. It may collide with a neighboring skill. A model or harness update may silently change all of the above.

None of those failures is visible to a document scanner.

That is the gap addressed by Evaluating Skills, Not Just Agents, an August 20, 2026 preprint introducing Agentic Continuous Evaluation of Skills, or ACES. The paper’s central move is differential: run the same task with and without the target skill while holding the model, harness, workspace, supporting skills, and scorer fixed. The paired difference is Skill Lift.

NVIDIA released the methodology in the experimental open-source SkillEvaluator, which combines structural and security gates with semantic analysis, evaluation-dataset generation, and live agent trials. The important idea is larger than the tool: a skill is a behavior-changing dependency and should be released with evidence in the same way code is released with tests.

This guide shows how to build that evidence without turning one composite score into another vibe.

A skill is an executable behavior package

A production skill usually contains more than prose:

skill/
  SKILL.md
  scripts/
  references/
  assets/
  evals/

The short description affects discovery. The instructions affect workflow. Scripts move deterministic work out of the model. References provide domain context. Examples shape interpretation. Tools named by the skill create authority and cost. Evaluation assets declare the behavior that should survive the next change.

That makes the skill analogous to a small application package with five quality surfaces:

1. artifact quality: structure, scope, clarity, metadata, and maintainability;
2. supply-chain safety: provenance, secrets, URLs, scripts, dependencies, and permissions;
3. discovery: whether the agent selects the skill for relevant requests and ignores it for irrelevant ones;
4. execution: whether the agent follows the workflow, invokes the right capabilities, and recovers from errors;
5. marginal value: whether the skill improves outcomes enough to justify its cost and risk.

Static scanning covers the first two partially. Live paired evaluation covers the last three.

Why static scores are not runtime evidence

ACES scanned 145 real skills from internal enterprise repositories and public catalogs. The paper reports:

• 94.5% passed the default structural gate;
• 86.2% passed the LLM-judge rubric;
• structural and judged scores had Spearman correlation of only 0.14;
• for the 62 production skills with matching scan and live metadata, the correlations between scan scores and live lift were statistically indistinguishable from zero.

The correct conclusion is not that scanning has no value. Structural checks are fast, deterministic, and useful for author feedback. Security scanners can catch leaked credentials, suspicious URLs, destructive commands, and prompt-injection markers before any live run. An LLM rubric can flag vague scope or incomplete error handling.

The conclusion is narrower: scanning a behavior-changing artifact is not the same as executing it.

The analogy in the paper is apt. Compiling with strict warnings can find many defects. It does not prove the program solves the user’s task.

Use tiers:

NVIDIA SkillEvaluator uses a related multi-tier design. Its repository labels live evaluation experimental and advisory by default unless the operator enables blocking. That is a sensible default for adoption, not the final standard for a consequential production skill.

Skill Lift isolates marginal value

Let a metric score a task under two conditions:

with_skill(task) - without_skill(task)

Across tasks, attempts, and configured metrics, ACES defines lift as the mean paired difference:

Skill Lift = mean(metric_with_skill - metric_baseline)

The design holds constant:

• user question;
• agent and model;
• harness version;
• workspace and fixtures;
• prerequisite and helper skills;
• decoy skills;
• grader and rubric;
• attempt policy.

Only the availability of the target skill changes.

An absolute score cannot tell whether the agent succeeded because the skill was good or because the model already knew the workflow. Lift can.

The paired design also reveals a useful future failure mode: as the baseline model improves, the skill’s marginal lift can shrink even while the absolute with-skill score rises. A skill that once encoded rare expertise may later become redundant overhead.

Build an author-owned task contract

The evaluation suite belongs with the skill. Do not begin with a generic benchmark generated from the document and never reviewed by the author.

For each case, declare:

{
  "id": "refund-investigation-implicit-01",
  "prompt": "A customer says the refund never arrived. Trace what happened and prepare the next safe action.",
  "expected_skill": "refund-investigation",
  "expected_script": "scripts/inspect_refund.ts",
  "ground_truth": {
    "refund_id": "rf_001",
    "status": "processor_pending",
    "next_action": "escalate_processor_case"
  },
  "expected_behavior": [
    "Read the refund-investigation skill before invoking its script.",
    "Use the supplied fixture account and do not query unrelated customers.",
    "Distinguish processor-pending from merchant-failed status.",
    "Do not issue a second refund.",
    "Cite the processor receipt in the final answer."
  ]
}

The behavior list is readable and flexible, but natural-language criteria require a judge. Replace high-consequence expectations with deterministic graders wherever possible:

• exact output schema;
• mock service state;
• permitted customer IDs;
• script and argument sequence;
• absence of a second refund;
• receipt hash included;
• no network egress outside the fixture environment.

Use natural-language judging for qualities such as explanation clarity or whether a recovery was sensible. Do not use it to decide whether money moved.

Cover four prompt buckets

The ACES workflow describes a useful minimum set:

1. Explicit: the user names the skill.
2. Implicit: the user describes the need without naming it.
3. Contextual: the request includes realistic noise or a larger goal.
4. Negative control: the request is adjacent or irrelevant and the skill should not activate.

These buckets separate content from routing.

An explicit test answers, “Does the workflow work when activation is easy?” An implicit test exercises the description. A contextual test exercises discrimination in real task language. A negative test measures over-triggering and unnecessary context or authority.

For a production skill, add four more buckets:

• error path: dependency missing, tool timeout, malformed input, partial output;
• security: untrusted content attempts to redirect the skill or expose a secret;
• conflict: another instruction surface or skill gives incompatible guidance;
• upgrade: the stable and candidate model–harness tuples run the same case.

Do not let generated cases define the contract. They may bootstrap coverage; the domain owner must review inputs, expected effects, and graders.

Test in isolation and in a realistic group

A skill can be excellent after activation and invisible in a crowded workspace.

Run two modes:

Isolation mode

Only the target skill is available. The result primarily measures procedural content: scripts, instructions, workflow order, and error handling.

Group mode

The target sits beside fixed prerequisite, helper, sibling, and decoy skills. The agent must discover and select correctly.

Keep the same non-target skills in both treatment and baseline conditions. Otherwise, the baseline becomes an empty workspace and the measured lift incorrectly includes the advantage of having any skill at all.

Compare:

routing premium = group lift - isolation lift

A negative routing premium suggests that names, descriptions, or overlap make the target hard to distinguish. A high isolation score cannot rescue a skill the agent never loads.

Grade the trajectory, not only the answer

ACES uses six default metrics:

The paper maps these into security, correctness, discoverability, effectiveness, and efficiency for stakeholder review. Keep the source metrics visible. A friendly dimension label should not hide why a score moved.

Normalize trajectories into a shared representation. ACES uses the Agent Trajectory Interchange Format, which records ordered messages, tool calls, observations, and metrics. ATIF v1.7 also supports embedded subagent trajectories.

At minimum, retain:

release manifest
available skill manifest
activation and file-read events
tool definitions and calls
tool observations and errors
workspace before/after diff
final response
token, cache, latency, and cost metrics
security and policy verdicts
subagent links
grader version and raw verdicts

Without the skill manifest, a passing trajectory cannot prove which version was available. Without the tool trace, a correct answer can hide an unsafe path. Without failed attempts, reliability looks artificially clean.

Do not use equal metric weights as policy

ACES uses equal one-sixth weights for its default composite as an inspectable diagnostic. The paper explicitly does not claim that every metric is equally causal or valuable.

Production policy should preserve hard constraints:

type SkillEvaluation = {
  security: number
  skillExecution: number
  skillEfficiency: number
  accuracy: number
  goalAccuracy: number
  behaviorCheck: number
  costUsd: number
  p95LatencyMs: number
  criticalViolations: number
}
 
function promote(candidate: SkillEvaluation, lift: SkillEvaluation): boolean {
  if (candidate.criticalViolations > 0) return false
  if (candidate.security < 1) return false
  if (lift.goalAccuracy < 0.05) return false
  if (lift.behaviorCheck < 0) return false
  if (lift.costUsd > 0.20) return false
  return true
}

The thresholds are workload policy, not universal defaults. A security metric should usually be a gate, not a weight that can be offset by a pleasant final answer.

Report at least three views:

• outcome lift: accuracy and goal achievement;
• process lift: activation, workflow, behavior, and efficiency;
• risk and economics: security, latency, tokens, tool cost, and external side effects.

Treat negative lift as a debugging asset

In the ACES study, composite lift was positive in 689 of 947 paired cases, zero in 171, and negative in 87. The negative cases are not noise to discard.

They can reveal:

• the skill was discovered but misused;
• instructions caused a truncated or meta-level answer;
• a required verification step disappeared;
• tool calls increased without improving the outcome;
• a script made error recovery worse;
• the skill conflicted with a sibling;
• the treatment run failed while the baseline completed;
• the task or grader itself is unstable.

Triage by mechanism:

The objective is not to maximize lift blindly. It is to understand which mechanism the skill changed.

Add domain graders for consequential skills

Generic metrics cannot verify every business effect. ACES supports Bring Your Own Task and Bring Your Own Grader paths; an internal harness should do the same.

Examples:

• a deployment skill should be graded on artifact, environment, rollout state, health checks, and rollback evidence;
• a finance skill should be graded on ledger entries, currency, accounting period, and approval binding;
• a compliance skill should be graded on source citations, jurisdiction, effective date, and unresolved ambiguity;
• a data-migration skill should be graded on row counts, checksums, schema constraints, and reversibility;
• a support skill should be graded on customer scope, policy eligibility, external effect, and receipt.

Domain graders should consume objective artifacts, not only the final response. A local run cited in the ACES paper found a synthetic secret canary remaining in an intermediate artifact even though the final file was repaired and generic outcome scores were perfect. Artifact-aware grading found what final-answer grading missed.

Put the skill change through CI

A pull request that changes a skill should produce a release evidence packet:

skill_release:
  id: refund-investigation
  candidate_version: 2.4.0
  previous_version: 2.3.0
  artifact_sha256: "..."
  release_tuple_sha256: "..."
  task_set_version: 8
  graders_version: 5
  conditions:
    - baseline_without_candidate
    - candidate_isolation
    - candidate_group
  attempts_per_case: 3
  evidence:
    static_scan: artifacts/static.json
    supply_chain: artifacts/security.json
    trajectories: artifacts/atif/
    paired_report: artifacts/lift.json
    domain_report: artifacts/domain.json
  decision:
    status: pending_review

Use a staged pipeline:

1. validate manifest and referenced files;
2. lint deterministic scripts;
3. scan secrets, Unicode, URLs, dependencies, and injection patterns;
4. check semantic overlap with installed skills;
5. generate or validate the paired environments;
6. run explicit, implicit, contextual, negative, error, security, and conflict cases;
7. normalize and retain all trajectories;
8. execute deterministic and judged graders;
9. compute per-case lift and uncertainty;
10. apply hard gates and require review for material authority changes;
11. canary the complete model–harness–skill release tuple;
12. retain rollback and revocation metadata.

Choose release gates that resist gaming

Gate on distributions and critical slices:

promotion:
  artifact:
    critical_findings: 0
    unresolved_secret_findings: 0
  live:
    min_goal_accuracy_lift: 0.05
    min_behavior_lift: 0.00
    min_negative_control_precision: 0.95
    max_security_regressions: 0
    max_domain_critical_failures: 0
    max_p95_latency_regression: 0.15
    max_cost_per_accepted_outcome_regression: 0.10
  evidence:
    min_paired_case_coverage: 0.95
    require_stable_and_candidate_runs: true
    require_trajectory_for_every_scored_case: true

Do not accept a higher average when a critical security or side-effect slice regresses. Do not silently exclude timeouts or failed treatment runs; report missing pairs and condition failures separately.

A four-week adoption plan

Week 1: select one consequential skill

Choose a skill with a clear task distribution, observable outputs, and meaningful failure cost. Write 12 author-reviewed cases across the four core prompt buckets and the main error paths.

Week 2: build paired worlds and graders

Make the baseline fair. Keep prerequisites and decoys fixed. Implement objective graders for scripts, tool arguments, artifacts, external mock state, and forbidden effects.

Week 3: capture portable trajectories

Export ordered messages, tool calls, observations, costs, workspace diffs, and the complete release manifest. Calibrate any judged behaviors against a human-reviewed subset.

Week 4: gate one real change

Run stable and candidate skill versions. Review negative lift by mechanism. Block critical regressions, canary the winner, and retain the evidence packet with the release.

Limits of the current evidence

The ACES results should be read with their scope attached:

• The 145-skill corpus is concentrated in system access, deployment, platform, and data-infrastructure work.
• Live coverage is uneven across four primary harnesses; 58 of 64 production skills contributed scored paired cases.
• The 947 cases are clustered by skill, harness, and repeated trial, not 947 independent skills. The paper reports cluster sensitivity checks, but broader replication remains useful.
• Some metrics use LLM or RAGAS judgment. Judge version and calibration affect absolute values.
• Missing or unreconstructable trajectories exist in the evidence inventory; paired headline results include cases where both conditions were scored.
• Mean lift is not a transferable constant. It depends on the model, harness, task set, workspace, skill neighborhood, and grader.
• SkillEvaluator is experimental and community-supported, with live model and managed sandbox costs.

These limitations argue for workload-specific evaluation, which is exactly what the paired method enables.

The release standard to aim for

A production skill should be promotable only when a reviewer can answer:

• What tasks is it supposed to improve?
• When should it activate—and when should it stay silent?
• Which scripts, tools, data, and identities can it use?
• What does it add beyond the baseline agent?
• Does it still add value among neighboring skills?
• Which trajectories prove discovery, workflow, and effects?
• Did security, latency, or cost regress?
• Which model–harness tuple was tested?
• How is the skill revoked or rolled back?

That is a much higher bar than “the markdown looks good.” It is also a practical one. Static review remains the fast first line; paired live evaluation supplies the missing runtime evidence.

What to read next

• Harness Engineering in August 2026: The Control Plane Gets Measured
• HarnessRisk in Practice: Turn the Agent Lifecycle Into a Security Test Matrix
• Your Agent Followed the Rule—or Did It?
• Harness Candidates Are Model Checkpoints
• Dataset-First Agent Engineering