NVIDIA SkillEvaluator: Agent Skills Guide

The short answer

NVIDIA SkillEvaluator is an open-source evaluation framework for AI-agent skills. It does not merely ask whether a SKILL.md file looks well written: it can validate the artifact, detect semantic overlap, generate task datasets, and compare an agent running with and without the skill inside an isolated environment.

For teams shipping reusable skills, the useful idea is Skill Lift: the measured difference between the same task run with a skill installed and the baseline run without it. NVIDIA’s August 19, 2026 benchmark report covered more than 300 verified skills across 30-plus NVIDIA products. In its published snapshot, average gains were 41 points for correctness, 40 for discoverability, 39 for effectiveness, and 35 for efficiency. Those are benchmark score differences—not percentage increases in production success rate—and NVIDIA labels the project experimental.

That distinction matters. SkillEvaluator can make a skill release more evidence-based, but it cannot prove that every agent, model, repository, or production workload will receive the same benefit.

What SkillEvaluator actually evaluates

A skill is a reusable package of instructions and supporting files that an agent can load for a specialized task. The evaluator treats it as a release artifact rather than as harmless prompt text.

Its three entry points answer different questions:

The tiers are independent entry points. A team can start with a keyless quality check before configuring a model provider or sandbox. That makes the framework practical for a staged CI rollout rather than an all-or-nothing benchmark project.

The evaluation pipeline

The important design choice is the controlled comparison. In a Tier 3 evaluation, the task prompt, model, inputs, grading criteria, and harness are held constant as far as the setup allows. The variable being tested is whether the skill is installed.

Figure 1 — An editorial rendering of the SkillEvaluator flow. It summarizes the workflow described in the official SkillEvaluator documentation; it is not an NVIDIA product screenshot.

Tier 1 focuses on failure modes that are easy to miss in a text review: malformed frontmatter, leaked personal information, missing or incompatible licensing, suspicious Unicode, unsafe scripts, and agent-oriented security issues. NVIDIA’s separate SkillSpector project supplies specialized security scanning in the Verified Skills pipeline.

Tier 2 addresses a different quality problem: a catalog can become noisy when multiple skills repeat the same instructions or activate for overlapping tasks. Similarity checks do not tell you whether a skill is malicious; they help identify catalog bloat and ambiguous ownership.

Tier 3 is the behavioral test. SkillEvaluator uses Harbor to package evaluation cases and run agents in isolated environments. A generated dataset can include explicit, implicit, contextual, and negative cases, so a useful evaluation should test both “use this skill when relevant” and “do not load or apply it when irrelevant.”

What NVIDIA’s benchmark says—and does not say

The August 12 benchmark snapshot reported in NVIDIA’s August 19 post showed these catalog-wide averages:

The strongest direct result is the rise in correctness and effectiveness on the evaluated specialized tasks. But the table should not be read as a universal pass rate. NVIDIA says most skills had one attempt per task, only 15% of published results had two attempts, and the report did not include confidence intervals. The catalog is also continuously evaluated, so the repository’s current benchmark data can differ from the snapshot used in the article.

The efficiency result needs extra care. Efficiency is a scored dimension, while token use and execution time are tracked separately. NVIDIA gives contrasting examples: jetson-optimize-memory reduced tokens and execution time in one single-attempt example, while cuopt-install increased both. A skill can improve task correctness while still making a workflow longer or more expensive.

This is why the benchmark is more useful as a release diagnostic than as a marketing number. It can show that a skill helps a defined task set, identify regressions, and expose optimization opportunities. It cannot replace workload-specific tests, human review, access controls, or production observability.

A practical quickstart for a skill repository

The official repository documents an offline starting point. Install the evaluator with uv, then run deterministic checks against a directory containing SKILL.md:

1uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
2skillevaluator validate ./my-skill \
3  --checks schema,pii,license,quality,unicode,lint \
4  --no-dedup

A smaller quality-only check is useful during authoring:

1skillevaluator quality-check ./my-skill

For a live comparison, first generate and review an evaluation set:

1skillevaluator create-eval-dataset ./my-skill --full
2skillevaluator tier3 evaluate ./my-skill \
3  --agents codex \
4  --env-mode docker

The exact provider and agent setup matters. The SkillEvaluator README says Tier 3 needs an evaluator provider for generation or grading, credentials for the selected agent, and a sandbox. Hosted model calls and managed sandboxes can incur charges; local mode avoids managed-sandbox charges but does not make hosted model inference free.

Before enabling a full gate, run a small, representative dataset. Treat generated cases as drafts: inspect expected outputs, add failure cases from real incidents, and pin the model, harness, and evaluator versions in CI. A benchmark without reproducible inputs is difficult to compare over time.

A conservative CI release policy

A production team can use the tiers progressively:

1. Authoring: run schema, quality, license, PII, Unicode, and script checks on every change.
2. Catalog review: run overlap checks before accepting a new skill or changing its trigger description.
3. Security review: require the full scanner set for skills that execute scripts, access files, call networks, or handle credentials.
4. Behavioral evaluation: maintain a small reviewed dataset with positive, negative, and adversarial tasks.
5. Release decision: publish the report, record the exact commit and environment, and require human approval for material changes.
6. Post-release monitoring: rerun the evaluation when the skill, model, agent harness, dependencies, or task contract changes.

NVIDIA’s broader Verified Skills documentation adds provenance, ownership, skill cards, detached signatures, and release verification. These controls solve different problems from Skill Lift. A signed skill can still be unhelpful; a high-lift skill can still be unsafe if its source or permissions are not reviewed.

For agents that can touch databases, shells, cloud APIs, or private documents, keep runtime authorization outside the skill. The skill should explain a capability; the platform should enforce what the agent may actually execute. This separation is especially important when an agent can load skills from multiple registries.

How this fits with other agent skill systems

NVIDIA’s catalog is designed around the open Agent Skills model and can be installed into several compatible clients through the skills CLI. The NVIDIA skills repository documents targets including Claude Code, Codex, Cursor, Kiro CLI, and other compatible agents. Its catalog is mirrored from product-owned repositories, and skills are added or revised continuously.

That portability is valuable, but it also makes evaluation context-sensitive. A skill can be well designed for one harness and less effective in another because system prompts, context handling, tool adapters, and default behaviors differ. NVIDIA’s benchmark found a smaller average difference between Claude Code and Codex than between product domains, but teams should still test the client they deploy.

The same principle appears in the Hermes Agent skills system: skills are loaded on demand, use progressive disclosure, and can be created or installed through a hub. The operational question is not only “is this instruction reusable?” It is also “when does it activate, what can it access, and how do we verify a changed copy?”

For a broader architecture, compare this release-gate approach with the OpenAI Agents SDK sandbox and harness patterns and the Google ADK zero-trust guidance. Those links are contextual starting points, not substitutes for the current vendor documentation.

Common mistakes and debugging clues

Treating Skill Lift as a percentage

A +41 result means a 41-point difference on the report’s score scale. It does not mean the skill increased production success by 41%.

Comparing changing environments

If the model, harness, tool versions, task data, or grading rubric changes between runs, the result is no longer a clean skill comparison. Store those inputs with the report.

Running live evaluations on untrusted code

NVIDIA’s documentation recommends a Docker or cloud sandbox for untrusted code. Local execution is convenient, but convenience is not isolation.

Assuming a security pass proves safety

Static checks can find known patterns and structural problems. They cannot prove that a skill is safe in every environment or that its dependencies are trustworthy. Keep human review and least-privilege runtime controls.

Using a tiny or biased dataset

A dataset with only the happy path can reward a skill that fails on ambiguous, irrelevant, or adversarial requests. Add negative cases and evaluate the boundaries explicitly.

Forgetting catalog overlap

Two skills with similar triggers can compete for attention or load redundant context. Run Tier 2 before expanding a catalog, and make trigger descriptions narrow enough to be useful.

FAQ

Is NVIDIA SkillEvaluator a benchmark for foundation models?

No. It evaluates agent artifacts, starting with skills, by measuring task behavior with and without a skill. It can use different agent harnesses and models, but its primary question is whether the skill changes the agent’s result on a defined task set.

Can I run it without an API key?

Yes, deterministic Tier 1 checks and the documented quality check can run without an API key. Tier 2 embeddings, LLM-backed checks, and live Tier 3 evaluations need additional providers or credentials.

Does a high Skill Lift mean a skill is ready for production?

No. It is one signal. Production readiness also requires security review, dependency and license checks, authorization design, reproducibility, cost limits, observability, and tests against real failure modes.

What should developers publish with a skill?

Publish the skill’s owner, purpose, activation boundaries, dependencies, supported clients, verification steps, evaluation dataset or task contract, benchmark commit, and known limitations. If signatures or a catalog are used, document how consumers verify them.

Conclusion

SkillEvaluator addresses a real gap in agent engineering: reusable skills are becoming a capability layer, but teams need evidence that a skill is safe, distinct, and useful. Its three-tier design gives developers a sensible path from offline validation to sandboxed behavioral evaluation, while the Skill Lift metric makes the contribution measurable.

The right adoption strategy is conservative. Start with deterministic gates, review the task dataset, run controlled comparisons in the target harness, and treat benchmark scores as evidence for a specific workload—not as a blanket guarantee. Pair the evaluator with signed provenance and runtime authorization, and agent skills become closer to testable software releases than opaque prompt bundles.

Sources and visual credits

• NVIDIA Technical Blog: Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator — benchmark methodology, results, limitations, and examples.
• NVIDIA SkillEvaluator on GitHub — open-source CLI, tiers, requirements, and support status.
• NVIDIA SkillEvaluator documentation — quickstart, reports, CI, and sandbox guidance.
• NVIDIA Verified Agent Skills documentation — provenance, scanning, signing, and skill-card model.
• NVIDIA skills catalog — installation targets and current catalog.
• Hermes Agent Skills System — independent implementation reference for progressive skill loading.

Visual credits: Figure 1 is an original Mermaid diagram by the author, based on the official NVIDIA SkillEvaluator documentation. The comparison tables are original editorial renderings of figures published by NVIDIA; readers should consult the primary source for scope and limitations.