SkillEvaluator

SkillEvaluator is an open-source command-line framework developed by NVIDIA for evaluating artifacts used by AI agents. Its first supported artifact type is Agent Skills, packages of instructions and supporting files that extend an agent. The framework combines static validation, semantic-overlap analysis, synthetic task generation, and live agent comparisons. It is written for Python 3.12 and 3.13 and distributed under the Apache License 2.0.^[1][2]

NVIDIA labels SkillEvaluator experimental. Its support policy promises community help on a best-effort basis but no service-level agreement or NVIDIA enterprise-support entitlement. This status distinguishes an available open-source tool from a production support commitment.^[13]

Key facts

Scroll sideways for more →

Purpose and evaluation model

SkillEvaluator separates three questions that are often collapsed into one model evaluation result. Tier 1 asks whether an artifact is structurally acceptable and whether static checks find policy or code problems. Tier 2 asks whether instructions repeat material within a skill or overlap another skill. Tier 3 asks whether a selected agent performs differently when the skill is available. The tiers are independent entry points: an operator can run one without first completing the others.^[1]

The combined skillevaluator validate command applies a gate policy on top of that modular design. Tier 1 always affects the exit code. Tier 2 is blocking when it runs unless --no-block-on-dedup makes it advisory. Tier 3 is advisory unless --block-on-agent-eval promotes it into the gate. This distinction matters because a three-tier report can contain useful live results even when those results do not decide the process exit status.^[5][7]

Scroll sideways for more →

Tier 1 validation

The default Tier 1 set checks schema, optional semantic version metadata, security, personally identifiable information, license evidence, code integrity, Unicode safety, artifact quality, and script lint. Dependency auditing is available but must be selected. The dependency check scans requirements files. When a pyproject.toml is present, pip-audit --local audits the active Python environment rather than resolving the dependencies declared by that file, and the audit may consult network-backed vulnerability data. The static security path can draw evidence from SkillSpector, Bandit, Semgrep, and Gitleaks, while pip-audit supports the dependency check. The base installation does not include all of these tools: some Python scanners come through the security extra, and Semgrep, SkillSpector, and Gitleaks remain separate executables.^[5][9]

Missing evidence is not treated as a clean result. If a required scanner is absent, times out, crashes, or returns malformed output, validation records INCOMPLETE and exits nonzero. Optional large language model analysis can add contextual security review or verify static findings, but it does not replace the scanners or turn unavailable evidence into a pass.^[5]

The default code-integrity check also has a narrow boundary. It does not import or execute target-controlled Python code. Its test discovery only counts regular in-tree files matching test-name patterns, and it explicitly reports that execution and coverage measurement did not occur. A Tier 1 pass therefore does not mean that an artifact's tests ran or that its executable content is safe at runtime.^[5]

quality-check produces a heuristic score from 0 to 100. Correctness has a 35 percent weight, discoverability 25 percent, reliability 25 percent, and efficiency 15 percent; the default passing mark is 70. These categories describe artifact quality signals, not the five task-performance dimensions used in Tier 3. The optional rubric-eval is another score: a model rates nine criteria from 0 to 10, while local code requires at least 7 on every criterion and an importance-weighted overall score at or above the configured minimum.^[5]

Tier 2 semantic comparison

Tier 2 has different methods for intra-skill and inter-skill comparison. context-optimization-check, also exposed as dedup-scan, divides one skill into chunks and embeds them. Pairs with cosine similarity of at least 0.80 become candidate clusters. A chat model then sees the candidate text and classifies it as a duplicate, intentional detail, or related but distinct material. A duplicate at confidence 0.70 or higher is normally a HIGH blocking finding; lower-confidence and short configuration-like cases remain advisory.^[6]

similarity-check compares skills across a collection using embeddings, without calling a chat model. Its classification boundaries are fixed: EXACT_DUPLICATE begins at 0.95, HIGH_SIMILARITY at 0.90, SIMILAR at 0.75, and LOOSELY_RELATED at 0.50. The first two are blocking severities and the next two are advisory. Changing the reporting threshold controls which results appear; it does not redefine the bands.^[6]

The comparison is semantic rather than proof of plagiarism or functional equivalence. Embedding models, chunking, thresholds, and short descriptions all influence results. Hosted embedding calls can transmit skill names, descriptions, or full text to the selected provider, and intra-skill verification sends candidate passages to a chat model. The framework supports a local OpenAI-compatible endpoint for operators who do not want this content to leave their environment.^[6][8]

Tier 3 live agent evaluation

Tier 3 runs actual agent command-line programs against evaluation cases. At the pinned main-branch snapshot it supports Codex, Claude Code, and OpenCode. SkillEvaluator stages the same case for a with-skill arm and, unless --skip-baseline is set, a without-skill arm. Harbor executes the trials in the selected environment. The intended experimental difference between the two arms is the presence of the skill.^[7][11]

An operator can author cases or generate a starting dataset with skillevaluator create-eval-dataset ./my-skill --full. The full generator creates explicit, implicit, and contextual-positive trigger cases plus a negative case that should not activate the skill. Each case contains a prompt, expected output, and optional assertions. NVIDIA's documentation tells users to review and edit generated cases, because synthetic cases are inputs to the experiment rather than ground truth discovered by the tool.^[7]

Before a run, doctor and health-check inspect the installation, selected provider, agent names, credentials, and execution backend. A representative live command is skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker. The agent CLI is supplied by the user; installing SkillEvaluator does not install Codex, Claude Code, or OpenCode.^[9][11]

Two credential roles normally remain separate. An evaluator provider generates datasets and grades standard trials. The live agent uses its native credential to perform the task. Both originate in the host environment. A checked-out skill cannot provide, alias, or redirect listed operator credentials through its evaluation configuration. NVIDIA Build has documented one-key bridge paths for the three agents in Docker or local mode, but other provider and backend combinations still need the agents' native credentials.^[7][11]

Providers and execution environments

The documented chat-provider choices include NVIDIA Build, OpenAI, Anthropic, Amazon Bedrock, and OpenAI-compatible endpoints. Tier 2 embeddings can use NVIDIA Build, OpenAI, or an OpenAI-compatible service. Anthropic and Bedrock do not supply embeddings in this integration, so using either as the chat evaluator requires a separate embedding provider for Tier 2.^[8]

Installation extras divide the dependency footprint. The base package is intended for static validation and does not pull in Harbor or public LLM clients. llm, tier2, tier3, and security add the corresponding Python dependencies; all combines Tier 2, Tier 3, and security. External scanner binaries, Docker, and the agent CLIs remain separate regardless of the extra selected.^[2][9]

Docker is the default Tier 3 backend and requires a running Docker daemon plus Docker Compose v2. The pinned documentation lists 16 environment values: Docker, local host execution, and 14 Harbor-native backends. Managed backends require their own packages and credentials. The same command can therefore represent materially different isolation, cost, and infrastructure conditions.^[11]

Local mode is experimental and is intended for trusted skills and workspaces. Linux uses Bubblewrap namespace isolation. macOS uses Seatbelt to restrict file and network activity, but the documentation calls it weaker because it cannot provide full process isolation. Native Windows local mode is hard-unsupported and fails before agent discovery; the documented alternatives are WSL2 or Docker. NVIDIA recommends Docker or a remote Harbor backend for untrusted code.^[11]

Reports and scoring

Tier 1 and Tier 2 can render CLI, JSON, HTML, and Markdown reports. Without an explicit report flag, a run shows a compact terminal view and writes HTML and JSON. Validation of a skill also writes BENCHMARK.md, a publication card whose overall state is PASS, FAIL, or INCOMPLETE. Missing trustworthy evidence from a required scanner produces INCOMPLETE; a publication recommendation appears only after a clean pass.^[10]

Tier 3 uses a timestamped results directory. It records result.json, the resolved run and attempt configuration, an HTML report, arm summaries, per-trial artifacts, and lift data when both arms have scores. Raw Harbor job directories are deleted after collection unless the operator requests retention. An unavailable metric remains null or appears as NO SCORE; it is not silently converted to zero.^[10]

With standard grading, reports organize signals into Security, Correctness, Discoverability, Effectiveness, and Efficiency. Each dimension is on a 0 to 1 scale. The framework's judge bands call 0.70 or above a pass, 0.40 through less than 0.70 neutral, and less than 0.40 a fail. Security in this table is an evaluation dimension, not a certificate that the artifact has no vulnerabilities.^[7][10]

Skill Lift is the with-skill score minus the without-skill score. Standalone reports call lift of at least +0.05 a pass, lift between -0.10 and +0.05 neutral, and lift at or below -0.10 a fail. The single Tier 3 verdict in combined validation is stricter on regressions, failing at -0.05 or below. Lift is absent when the baseline arm is skipped.^[10]

pass@k is reported separately from the dimension averages. With k attempts per case, a case counts as passed if at least one attempt meets the configured pass threshold. This measures repeatability within the selected cases and configuration. It does not make a single-case or single-model result universal, and the default one-attempt setting provides little information about run-to-run variance.^[7][10]

Relationship to NVIDIA Verified Skills

SkillEvaluator is part of NVIDIA's Verified Skills pipeline and public skills catalog. NVIDIA describes the pipeline as cataloging, scanning, evaluating, signing, and documenting submitted skills. SkillEvaluator can supply validation and live-evaluation evidence and produces the BENCHMARK.md card stored with a skill. The live evaluation compares the same tasks with and without the skill.^[1][14]

In this context, verified is a catalog process status. It is not independent certification that a skill is harmless, correct for every task, or beneficial with every agent and model. A signature can establish artifact provenance without reproducing the performance measurements, and a clean static scan can still miss behavior outside its rules.

Security and trust boundaries

SkillEvaluator treats the target and evaluation inputs as untrusted at several file boundaries. Staging rejects non-ignored symbolic links, special files, hard-linked files, path escapes, and device crossings. Selected evaluator files are read through bounded paths, while report writes are atomic and refuse symlink, junction, or parent-traversal redirection.^[10][12]

Provider secrets and other operator controls come from the host rather than the skill. Evaluation configuration cannot set or reference listed API keys, provider base URLs, AWS credentials, host launchers, Docker controls, or similar protected variables. On supported NVIDIA Build bridge paths, Docker transfers the provider secret through a host-only file with restrictive permissions; local bridge trials receive scoped capability tokens. Direct OpenCode is an exception because it calls NVIDIA Build itself and must receive the key.^[7][11]

These measures limit specific attacks but do not prove safe execution. Static scanners depend on their rules and versions. Model-based security review can be mistaken or manipulated. Local mode exposes a broader host boundary than a container. A custom Dockerfile or base image remains trusted executable input rather than becoming a security boundary. The security policy also excludes hostile same-user concurrent filesystem mutation from its staging guarantee, because repeated validation is not one coherent filesystem snapshot.^[7][11][12]

Release history

SkillEvaluator's release and branch histories diverged shortly after publication.^[3][4]

Scroll sideways for more →

The v0.1.0 notes describe the three-tier pipeline, provider support, Harbor-backed agent paths, semantic-version validation, removal of implicit host-side pytest execution, credential isolation, and macOS Seatbelt hardening. The later main-branch changelog records further gating, reporting, and staging-security changes. Because no v0.2.0 tag or GitHub release existed at the research cutoff, 0.2.0 should not be called the latest released version. Installing from an unpinned main branch also does not reproduce the tagged v0.1.0 package.^[3][4]

Vendor benchmark claim

On Aug. 19, 2026, NVIDIA AI said that it had benchmarked 300-plus NVIDIA-verified skills and that, across those benchmarks, access to the skills raised correctness by 41 points, effectiveness by 39 points, and efficiency by 35 points. The social post did not identify the benchmark snapshot, filtering rules, aggregation unit, missing-result treatment, or rounding method.^[17]

The pinned Aug. 17 public benchmarks.json revision contains 341 skill directories and 3,180 flattened result rows. Of those skills, 321 have results and 20 do not; each of the five reported dimensions has 636 rows. A direct equal-row arithmetic mean gives +40.0660 for Correctness, +39.3003 for Effectiveness, and +35.6619 for Efficiency. Ordinary nearest-integer rounding would produce 40, 39, and 36 rather than the advertised 41, 39, and 35.^[15]

The public aggregation script extracts dimension tables from per-skill BENCHMARK.md files into flat rows. It does not publish a second aggregation rule that explains the social headline. A different snapshot, skill-level weighting, filter, or rounding convention could account for the difference, but none is specified in the cited public materials. The three advertised figures are therefore vendor-reported and not transparently reproduced by the obvious calculation; they are not evidence of independent validation.^[15][16]

Reproducibility and limitations

Reproducing a SkillEvaluator result requires more than preserving the final report. A useful record includes the SkillEvaluator tag or commit, skill and evaluation-dataset revisions, validation profile, scanner versions, agent CLI and model, evaluator model and provider, execution backend, grading mode, thresholds, attempt count, concurrency, and whether a baseline arm ran. Hosted models and managed sandboxes can change independently of the source repository, while live model calls can incur cost.^[7][8][10]

Synthetic cases measure the task distribution they encode. Agent runs also vary with model sampling, tool availability, environment state, and attempt count. A positive Skill Lift is evidence about the paired run, not a general percentage increase for all uses of a skill. Static quality and security scores have different meanings from live task scores and should not be averaged as if they were interchangeable.

Academic agent benchmarks illustrate the broader need for interactive evaluation. AgentBench, published at ICLR 2024, tested 29 models in eight environments and reported persistent failures in long-horizon reasoning, decision-making, and instruction following. That study does not test SkillEvaluator, but it shows why agent evaluation needs environments and multi-step tasks rather than only text benchmarks.^[18]

LLM judging adds a separate validity problem. Zheng and colleagues found that a strong judge could exceed 80 percent agreement with human preferences in their chat-assistant setting, but documented position, verbosity, self-enhancement, and limited-reasoning biases. Ye and colleagues studied 12 potential judge biases and found significant bias in some tasks. These results support human review and repeated trials; they do not validate SkillEvaluator's specific judge prompts, dimensions, or thresholds.^[19][20]

No substantive independent audit, peer-reviewed study, or third-party reproduction of SkillEvaluator itself was located by Aug. 20, 2026. The available performance artifacts were produced within NVIDIA's own skills pipeline. The framework is therefore best understood as an experimental evaluation toolkit with explicit controls and report contracts, not as an independently validated standard.

See also

• LLM-as-a-judge
• Benchmark

References

1. ^NVIDIA. "SkillEvaluator README." Pinned commit `1cec101`. Aug. 20, 2026. github.com/...README.md
2. ^NVIDIA. "SkillEvaluator package metadata." Pinned commit `1cec101`. Aug. 20, 2026. github.com/...pyproject.toml
3. ^NVIDIA. "SkillEvaluator 0.1.0: Initial Public Release." Aug. 5, 2026. github.com/...v0.1.0
4. ^NVIDIA. "SkillEvaluator changelog." Pinned commit `1cec101`. Aug. 20, 2026. github.com/...CHANGELOG.md
5. ^NVIDIA. "Tier 1: Validation." SkillEvaluator documentation, pinned commit `1cec101`. Aug. 20, 2026. github.com/...tier1-validation.mdx
6. ^NVIDIA. "Tier 2: Deduplication." SkillEvaluator documentation, pinned commit `1cec101`. Aug. 20, 2026. github.com/...tier2-deduplication.mdx
7. ^NVIDIA. "Tier 3: Live Agent Evaluation." SkillEvaluator documentation, pinned commit `1cec101`. Aug. 20, 2026. github.com/...tier3-live-evaluation.mdx
8. ^NVIDIA. "Providers and Credentials." SkillEvaluator documentation, pinned commit `1cec101`. Aug. 20, 2026. github.com/...configuration.mdx
9. ^NVIDIA. "Installation." SkillEvaluator documentation, pinned commit `1cec101`. Aug. 20, 2026. github.com/...installation.mdx
10. ^NVIDIA. "Reports and Results." SkillEvaluator documentation, pinned commit `1cec101`. Aug. 20, 2026. github.com/...reports.mdx
11. ^NVIDIA. "Agents and Sandboxes." SkillEvaluator documentation, pinned commit `1cec101`. Aug. 20, 2026. github.com/...agents-and-sandboxes.mdx
12. ^NVIDIA. "SkillEvaluator security policy." Pinned commit `1cec101`. Aug. 20, 2026. github.com/...SECURITY.md
13. ^NVIDIA. "SkillEvaluator support policy." Pinned commit `1cec101`. Aug. 20, 2026. github.com/...SUPPORT.md
14. ^NVIDIA. "NVIDIA Verified Skills." Accessed Aug. 20, 2026. docs.nvidia.com/skills
15. ^NVIDIA. "NVIDIA Skills benchmark results." Pinned commit `20bb6aa`. Aug. 17, 2026. github.com/...benchmarks.json
16. ^NVIDIA. "Aggregate benchmark results script." Pinned commit `20bb6aa`. Aug. 17, 2026. github.com/...aggregate_benchmarks.py
17. ^NVIDIA AI. "We benchmarked 300+ NVIDIA verified skills." X, Aug. 19, 2026. x.com/...2090113635683340622
18. ^Liu, Xiao, et al. "AgentBench: Evaluating LLMs as Agents." International Conference on Learning Representations, 2024. proceedings.iclr.cc/...b7e9f57-Abstract-Conference
19. ^Zheng, Lianmin, et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Advances in Neural Information Processing Systems 36, 2023. proceedings.neurips.cc/...-Datasets_and_Benchmarks
20. ^Ye, Jiayi, et al. "Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge." arXiv:2410.02736, 2024. arxiv.org/...2410.02736