NVIDIA NeMo Switchyard

NVIDIA NeMo Switchyard is an open-source proxy and Rust library for routing requests among configured large language models. Developed within NVIDIA's NeMo organization, it can present one client-facing model name while selecting a backend according to a fixed split, an LLM judge, conversation state, or signals from an agent's tool activity. It also translates among OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages formats.^[1][2][3]

Switchyard is infrastructure, not a language model. It contains no model weights and does not assign permanent rankings to models. A deployer supplies the endpoints, models, credentials, routing policy, and quality criteria. The first public standalone artifacts appeared on June 30, 2026, when versions 0.0.1 and 0.1.0 were published; NVIDIA described 0.1.0 as the first public release. Version 0.2.0, released on August 10, replaced much of the original Python routing path with a native Rust server and embeddable routing core.^[1][4][5]

Scroll sideways for more →

History and scope

The Switchyard name appeared in NeMo Platform 0.1.0 release notes on May 20, 2026. That local-first platform used Switchyard middleware in its inference gateway for random strong and weak model splits and cross-format translation. This use predates the public standalone package, but it was one component of the broader platform release.^[6]

The 0.1.0 release described itself as Switchyard's first public release. It provided a Python proxy, YAML route bundles, initial routing strategies, coding-agent launchers, and request metrics. Version 0.2.0 followed 193 commits later with a new Rust-centered architecture, explicit TOML deployment files, expanded translation, more routing policies, and separately published crates. It deprecated the earlier Python server and YAML profile system for new deployments.^[1][5]

NVIDIA's August 14 X post described Switchyard as a new open-source model-routing library and paired frontier models with Nemotron Lightning as an example. It was a promotion rather than the initial launch: it appeared four days after 0.2.0 and more than six weeks after the first public package.^[1][7] The model pairing in that post is an example, not a required Switchyard configuration.

Architecture

Switchyard sits between clients such as AI agents, SDKs, or applications and one or more model backends. The native server decodes an incoming request into provider-neutral types, runs a routing algorithm, calls or identifies a target, and translates the response or stream into the format expected by the client.^[2][3]

A version 0.2.0 TOML deployment separates three kinds of object:

Scroll sideways for more →

The separation lets several routes reuse a target and several targets share a provider connection. Credentials are named by environment variable rather than placed directly in the TOML file. Upstream formats are explicit; the server does not probe a provider and guess its API.^[3]

The standalone server exposes the OpenAI API-compatible /v1/chat/completions and /v1/responses paths, Anthropic API-compatible messages and token-counting paths, a model list, health checks, Prometheus metrics, JSON statistics, and optional routing logs. The translation layer handles buffered and streaming responses, tool definitions and results, reasoning and final-output items, and usage and cache metadata. Compatibility still depends on the fields accepted by the chosen provider.^[1]

Version 0.2.0 divides the implementation into focused Rust crates. switchyard-protocol defines normalized request, response, stream, tool, usage, error, and metadata types. switchyard-translation converts supported API formats. switchyard-libsy contains routing algorithms without requiring an HTTP server or provider SDK. switchyard-llm-client adds translated HTTP calls, and switchyard-server composes the pieces into the reference proxy.^[1]

Libsy can be embedded in another Rust host. Its algorithm yields requested model calls back to that host, which may execute them using its own transport, credentials, and retry logic. The packaged Python command instead starts the Rust server through a binding and configures supported coding agents, including Claude Code, OpenAI Codex, and OpenClaw, to use it.^[2][3]

Routing strategies

The native server's route types cover fixed traffic, experiments, pre-request classification, and multi-turn agent behavior.^[9]

Scroll sideways for more →

Capability-mode classification asks a judge for structured fields including an estimated probability that the weak target can complete the task. A deterministic policy applies base_threshold and threshold_step after the verdict. Invalid, inconsistent, or unparseable verdicts fall back to the strong target. Custom mode can validate a user-supplied JSON schema and select among more than two configured targets.^[10]

Escalation evaluates work after the weak model has produced a buffered reply. Consecutive escalation verdicts build a streak. When the configured confirmation count is reached, Switchyard discards that weak reply, calls the strong target, and pins later requests in the session to it. Before the latch, each turn waits for both the weak model and judge. This adds latency and changes streaming behavior, so suitability for one-shot or latency-sensitive traffic requires workload-specific testing.^[11]

The stage router works in both directions at the turn level. Severe errors, repeated unproductive activity, and prolonged exploration favor a capable target. Recent writes, edits, and settled progress favor an efficient one. Its confidence threshold determines when those signals override the configured default. The labels strong, weak, capable, and efficient are local roles, not properties that Switchyard discovers or guarantees.^[3][12]

NVIDIA's launch-era technical blog also described a learned prefill-activation router. Related research trains a network on residual-stream features to estimate which candidate model will answer correctly.^[8][18] That approach was research-stage at the 0.2.0 cutoff and is not one of the route types exposed by the tagged native server.^[9]

Installation and integrations

The Python distribution is named nemo-switchyard. Version 0.2.0 requires Python 3.12 or later and publishes a source archive plus wheels for macOS, Linux, and Windows on the listed architectures. The separate switchyard-server binary can be installed with Cargo. Both distributions use the Apache 2.0 license.^[1][4]

The proxy itself performs routing, translation, and telemetry rather than model inference. It can call hosted providers or self-hosted endpoints such as vLLM, NVIDIA NIM, Ollama, and other OpenAI-compatible services. Switchyard therefore has no general GPU requirement of its own, while a locally hosted target may require substantial accelerator memory and provider-specific serving software.^[2]

NeMo Platform and NeMo Relay use different integration boundaries. Platform 0.1.0 included Switchyard middleware inside its inference gateway.^[6] Relay 0.6.0 documents an experimental optional plugin that asks a separately running Switchyard Decision API for a target, then keeps credentials, provider binding, dispatch, retry, and fallback inside Relay. NVIDIA excludes that plugin from default Relay builds and pins its compatibility test to a separate branch and commit. The Relay plugin's early-access status does not change the standalone package into a stable Relay component.^[17]

Evaluation

The version 0.2.0 repository includes a Harbor-based benchmark harness for direct and routed comparisons. It can prepare closed-book variants of TB Lite, Terminal-Bench 2.0 and 2.1, and SWE-Bench Pro, pin agent versions, record a run manifest and server configuration, and collect final routing statistics. The tagged repository supplies procedures and example configurations, but it does not publish a canonical aggregate result table or committed completed runs. Reproduction requires access to the named model providers and careful pinning of models, prices, dataset, agent, book mode, concurrency, and retries.^[13]

LangChain published a partner evaluation using 145 controlled, multi-step tasks that averaged 6.3 model calls. The tasks covered customer support, incident investigation, workflow automation, tool use, retrieval, filesystem operations, and long-context summarization. Its escalation route used Nemotron 3.5 Lightning as the efficient model, Claude Opus 4.8 as the frontier model, Gemini 3.1 Flash Lite as judge, and two consecutive negative verdicts before escalation.^[14]

Scroll sideways for more →

Across five routed runs, LangChain reported that 7% of non-judge calls went to Opus. Relative to the Opus-only arm, routing reduced cost by 74% and accuracy by six percentage points. Judge calls accounted for 21.2% of routed spend, and the routed run cost ranged from $2.16 to $3.61.^[14]

The report includes important limits. The suite was saturated, with only 8.3 points between the two single-model arms. The routed arm's 2.3-point advantage over the efficient-only arm was smaller than observed run-to-run variation, so LangChain did not claim that routing beat the efficient model. It did not benchmark the stage router, and it warned that the result measured one workload rather than forecasting another.^[14]

A separate DevelopersIO hands-on test illustrates classifier sensitivity. In that practitioner's private coding workload, changing the judge model shifted weak-tier selections across 50 stored conversations from 39 to one. An eight-model, two-prompt sweep produced weak-selection rates from zero to 100 percent, and one capability configuration sent all 14 tested algorithmic programming problems to the weak tier. These observations are not a standardized benchmark, but they show why the judge, prompt, thresholds, and workload need local calibration.^[15]

Research context

Model routing predates Switchyard. RouteLLM, published at ICLR 2025, learned routers from human preference data and reported lower cost at comparable quality in its tested settings.^[19] Switchyard 0.2.0 removed an earlier RouteLLM integration, so the paper should be treated as related research rather than its implementation basis.^[1]

A broader 2026 study, LLMRouterBench, evaluated representative routing methods on more than 400,000 instances from 21 datasets and 33 models. It confirmed that model strengths can be complementary, but found that many routers performed similarly under unified testing, some did not reliably beat a simple baseline, and a substantial gap to oracle selection remained.^[20] This makes per-workload evaluation more informative than assuming that any router or model pair will produce a universal cost-quality improvement.

Maturity and limitations

Switchyard 0.2.0 is early-stage software. Its release notes call it pre-alpha and warn that APIs and configuration may change before a stable release; PyPI applies the less restrictive Alpha classifier. Neither label establishes production readiness.^[1][4]

The tagged known-issues document identifies five concrete defects:^[16]

• Buffered upstream work can continue after a client disconnect and still incur provider cost.
• Some classifier, escalation, and stage-fallback paths lack complete target attribution in statistics and metrics.
• The retry-recovery counter can stay at zero after a successful retry.
• Native session statistics omit the x-switchyard-session-id value.
• The native server does not send the documented X-Switchyard-Version header upstream.

Other tradeoffs follow from the routing design. Classifier calls add latency and cost. Escalation can duplicate work, while an incorrect decision can send a request to a model that does not meet the required quality or policy. Session affinity in 0.2.0 is process-local unless a host supplies another persistence mechanism. Protocol translation also cannot remove behavioral differences among providers, models, context windows, tool schemas, or streaming implementations.^[3][10][11]

Switchyard's documented role is routing, translation, fallback, and operational measurement. It is not a content-safety evaluator or a guarantee that heterogeneous targets apply equivalent policies. A deployment should test each route and fallback with its own quality, security, privacy, latency, and failure requirements, pin the software and model versions used for evaluation, and monitor judge and target behavior after changes.

References

1. ^NVIDIA-NeMo. Switchyard v0.2.0 release. GitHub, August 10, 2026.
2. ^NVIDIA-NeMo. Switchyard repository and README, version 0.2.0. GitHub, 2026.
3. ^NVIDIA-NeMo. Switchyard core concepts, version 0.2.0. GitHub, 2026.
4. ^Python Package Index. nemo-switchyard 0.2.0. August 10, 2026.
5. ^NVIDIA-NeMo. Switchyard v0.1.0 release. GitHub, June 30, 2026.
6. ^NVIDIA. NeMo Platform v0.1.0 release notes. May 20, 2026.
7. ^NVIDIA AI. NeMo Switchyard model-routing announcement. X, August 14, 2026.
8. ^Varshney, Tanay; Alexiuk, Chris; Agarwal, Ayush; Surla, Annie; Chung, Moon. Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard. NVIDIA Technical Blog, August 11, 2026.
9. ^NVIDIA-NeMo. Routing overview, Switchyard 0.2.0. GitHub, 2026.
10. ^NVIDIA-NeMo. LLM classifier routing, Switchyard 0.2.0. GitHub, 2026.
11. ^NVIDIA-NeMo. Escalation-router routing, Switchyard 0.2.0. GitHub, 2026.
12. ^NVIDIA-NeMo. Stage-router routing, Switchyard 0.2.0. GitHub, 2026.
13. ^NVIDIA-NeMo. Harbor benchmark guide, Switchyard 0.2.0. GitHub, 2026.
14. ^LangChain. How many of your agent's calls actually need a frontier model?. August 11, 2026.
15. ^Morishige, Hiroshi. Hands-on with the Rust-based NeMo Switchyard v0.2.0. DevelopersIO, August 12, 2026.
16. ^NVIDIA-NeMo. Known issues in Switchyard 0.2.0. GitHub, 2026.
17. ^NVIDIA. Switchyard integration in NeMo Relay 0.6.0. 2026.
18. ^Varshney, Tanay; Surla, Annie; Xu, Michelle; et al. LLM Router: Rethinking Routing with Prefill Activations. arXiv:2603.20895, version 3, August 11, 2026.
19. ^Ong, Isaac; Almahairi, Amjad; Wu, Vincent; et al. RouteLLM: Learning to Route LLMs with Preference Data. ICLR 2025.
20. ^Li, Hao; Zhang, Yiqun; Guo, Zhaoyan; et al. LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. Findings of ACL 2026.