NVIDIA Mission Control

NVIDIA Mission Control is data center operations and orchestration software from NVIDIA for what the company calls AI factories: large clusters built from its DGX systems and Grace Blackwell rack-scale systems. It bundles cluster provisioning, workload scheduling through Slurm and Run:ai on Kubernetes, monitoring dashboards, power management and an "autonomous recovery engine" that detects failed hardware, moves it out of service and restarts affected jobs.^[1][4][7] NVIDIA announced Mission Control at GTC on 18 March 2025, initially for Blackwell-based DGX systems.^[4][5] Version 2.5, which NVIDIA's technical blog announced on 26 October 2026, extends the recovery engine from Slurm to Kubernetes workloads managed by Run:ai and adds early access support for Vera Rubin NVL72 systems.^[1][2][3]

Overview

NVIDIA describes Mission Control as an integrated platform that packages its own operational practices and cluster automation into a single control plane.^[8] The product page says it covers the AI factory from developer workload scheduling and orchestration through monitoring and autonomous recovery, and that it "powers NVIDIA Blackwell and NVIDIA Rubin data centers".^[7] NVIDIA positions it alongside NVIDIA AI Enterprise: Mission Control handles infrastructure and workload management, while AI Enterprise supplies developer tools and frameworks that run on top of it.^[7]

NVIDIA's documentation hub calls Mission Control the recommended software for DGX B200, DGX B300 and Rubin NVL8 systems and for GB200, GB300 and Vera Rubin NVL72 systems.^[8] Release 2.5 supports DGX B200 and B300, DGX GB200 and GB300, and GB200 and GB300 NVL72 racks from other system makers, with Vera Rubin NVL72 in early access.^[2] NVIDIA's documentation hub lists the products included with a Mission Control purchase as NVIDIA Base Command Manager (BCM) for cluster management and provisioning, NVIDIA Run:ai for workload management and GPU orchestration, NVIDIA Unified Fabric Manager (UFM) for InfiniBand fabrics, and NVIDIA NetQ for NVLink and Ethernet observability, plus Mission Control-only capabilities delivered through Base Command.^[8]

History

Launch at GTC 2025

NVIDIA introduced Mission Control at its GTC conference on 18 March 2025, in the same announcement as the Blackwell Ultra DGX SuperPOD with DGX GB300 and DGX B300 systems. The press release called it "AI data center operation and orchestration software for Blackwell-based DGX systems".^[5] A companion post by Charlie Boyle on NVIDIA's corporate blog described it as "the only unified operations and orchestration software platform that automates the complex management of AI data centers and workloads", a vendor characterization.^[4] That post claimed the included Run:ai technology could raise infrastructure utilization by up to 5x and that the autonomous recovery features, "supported by rapid checkpointing and automated tiered restart features", could deliver up to 10x faster job recovery than manual methods. The post does not describe how either figure was measured.^[4]

The launch feature list included cluster setup and provisioning with inventory management, Slurm and Kubernetes workload orchestration, energy-optimized power profiles, autonomous job recovery, customizable dashboards, on-demand health checks, and integration with building management systems for power and cooling events, including leak detection.^[4] At announcement, NVIDIA said Mission Control was available for DGX GB200 and DGX B200 systems, that GB200 NVL72 systems with Mission Control were expected from Dell Technologies, HPE, Lenovo and Supermicro, and that support for DGX GB300, DGX B300 and GB300 NVL72 systems was expected later in 2025.^[4] NVIDIA also said Mission Control could be used with NVIDIA Instant AI Factory, a managed service offering Blackwell Ultra DGX SuperPODs in Equinix data centers.^[4][5] StorageReview reported the launch together with the partner plans and availability dates, and DataCentreNews UK also covered the announcement.^[19][20]

A technical post by Pradyumna Desale on 25 March 2025 described the initial architecture: a control plane that could manage an NVIDIA DGX SuperPOD containing both DGX B200 and DGX GB200 systems, provisioning built on Base Command Manager with rack management and leak detection policies for GB200 NVL72, and a telemetry stack built on UFM and NVIDIA NMX Manager that collected data from GPUs, Spectrum-X Ethernet and Quantum InfiniBand switches, and NVLink switches.^[6] It said validation and diagnostics were built on the testing framework NVIDIA used for its own supercomputers, such as Selene and Eos.^[6]

Release history

NVIDIA's documentation records the following releases. NVIDIA does not publish release dates in the release notes themselves, so dates are given only where a dated NVIDIA source exists.

NVIDIA's documentation hub lists 2.3.1 as the version previous to 2.5.^[8]

Mission Control 3.0 plans

In an April 2026 technical blog post tagged to GTC 2026, Pradyumna Desale and Casey Dugas described Mission Control 3.0. According to NVIDIA, 3.0 introduces a layered, API-driven architecture of modular services in place of "previously tightly coupled stacks that required synchronized releases", so that OEMs and independent software vendors can integrate Mission Control capabilities into their own stacks.^[18] The post also described multi-organization isolation, with Mission Control services running on KVM virtual machines and shared network fabrics segmented with VXLAN on Spectrum-X Ethernet and partition keys on Quantum InfiniBand; a domain power service that makes power a scheduling input for Slurm and Run:ai workloads; and integration with NVIDIA AIOps Collector and Platform Stacks (NACPS) for predictive anomaly detection.^[18] NVIDIA wrote that in one deployment its MAX-Q power profile let a data center run at 85% power with only 7% throughput loss.^[18] The post closed by telling readers to "stay tuned" for 3.0 release notes and implementation guides; as of 30 October 2026 the newest release in NVIDIA's release notes is 2.5.0.^[2][18]

Architecture and components

Mission Control is assembled from several NVIDIA products plus Mission Control-specific services. The 2.5 release notes group them as follows.^[2]

Base Command Manager is the provisioning base. BCM's administrator documentation describes Mission Control features as "extended features that are only available in the NVIDIA Mission Control edition" of BCM, unlocked by a license. On a cluster without that edition, BCM reports features such as leak detection, rack power, autonomous hardware recovery, autonomous job recovery, GB200 firmware flashing, building management integration, Run:ai, IMEX, NMX and power reservation steering as disabled.^[9] Since BCM 11.32, some Mission Control installers (NetQ, autonomous job recovery, autonomous hardware recovery and the power service) ship as plugins so they can be updated independently of BCM releases.^[9]

The control plane is a set of dedicated management nodes. For version 2.5, NVIDIA's installation guide lists Slurm login nodes, an admin Kubernetes cluster and a user Kubernetes cluster, in a highly available configuration; on GB200 and GB300 NVL72 systems it also requires all nine NVLink switch trays and 18 compute trays per rack to be provisioned before installation.^[17] The release notes list 10 control plane nodes for each supported platform, x86-only for DGX B200 and B300, with Arm required for BCM head nodes and Slurm nodes on DGX GB200, DGX GB300 and DGX Vera Rubin NVL72 except where x86 head nodes are permitted.^[2]

Autonomous recovery engine

The autonomous recovery engine (also called the Autonomous Resiliency Engine, or ARE, in release notes) is Mission Control's fault-handling system. Release 2.0.0 introduced it as a framework combining three parts: autonomous job recovery, autonomous hardware recovery, and the NVIDIA Resiliency Extension (NVRx).^[11]

• Autonomous job recovery watches running training jobs. In 2.0.0 its features included root cause analysis called Fault Attribution and Characterization (FACT), automatic job resumption with problem nodes isolated, standardized training logs, and a recovery dashboard NVIDIA calls the "cockpit UI".^[11]
• Autonomous hardware recovery handles the infrastructure side: failover with backups, cancelling runs when hardware faults are detected, and break/fix workflows for GB200 NVL72 compute trays and NVLink switch trays, with parallel remediation across racks.^[11] NVIDIA's March 2025 post said its workflow engine runs automated recovery playbooks and, when a component cannot be repaired, flags it for return merchandise authorization and can open a ticket with NVIDIA Enterprise Support.^[6]
• NVRx is an open-source Python package for PyTorch-based training that provides fault tolerance with hang detection, asynchronous and local checkpointing, and straggler detection.^[21] Mission Control's release notes state that NVRx is not installed with Mission Control and must be installed separately.^[2]

Before version 2.5, according to NVIDIA's October 2026 post, the recovery engine was limited to HPC workloads scheduled with Slurm.^[1] NVIDIA's March 2025 description had already said the engine "leverages NVIDIA Run:ai for Kubernetes to manage workloads".^[6]

Kubernetes support in Mission Control 2.5

Mission Control 2.5 extends the engine to Kubernetes workloads managed by Run:ai. NVIDIA's technical blog post by Ryan Prout and Casey Dugas describes the result as a three-layer control loop.^[1]

NVIDIA argues that keeping these roles separate is what makes the design portable across schedulers: the job service does not repair hardware, and the hardware service does not need to understand every training framework.^[1] The post walks through three cases. When a GPU fails during distributed training (showing up as ECC errors, a hung worker or a communication timeout), the job is restarted or rescheduled on healthy nodes while the hardware service isolates the suspect node and starts diagnostics in parallel. When a failure looks transient, such as a worker exit or a recoverable timeout on a healthy node, the job is retried under scheduler policy without quarantining hardware. When a node shows repeated GPU health warnings or connectivity problems before any job fails, the hardware service removes it from scheduling proactively.^[1]

The 2.5 release notes give the mechanism in more detail: the engine is integrated with the Run:ai KAI Scheduler when jobs use the Kubeflow PyTorch training operators. Mission Control monitors job output, correlates system logs and telemetry with failures to decide whether the cause is hardware or software, then requeues the job with its specification updated to exclude the nodes implicated in the most recent anomaly.^[2] The same release added support for NVRx 0.6.0 asynchronous checkpointing and a training telemetry client library that emits throughput, checkpoint and hang-detection log messages for the job recovery service to monitor.^[2]

Mission Control 2.5

NVIDIA announced Mission Control 2.5 in a technical blog post dated 26 October 2026, and its @NVIDIAAIInfra account promoted the release on X on 29 October.^[1][3] Besides Kubernetes support in the recovery engine, the post listed a new "AI-enabled network anomaly detection service (beta only)", central Keycloak authentication across services, more security and Kubernetes hardening, and early access for Vera Rubin NVL72.^[1]

The release notes add these details:^[2]

• Vera Rubin NVL72: early access is limited to "essential bring-up support" with Base Command Manager. Run:ai and the Autonomous Resiliency Engine are not supported on Vera Rubin NVL72 in 2.5. BCM 11.34.0 adds Vera Rubin NVL72 Grafana dashboards, alerts and Redfish telemetry, plus support for DGX OS 8.0.0 and Ubuntu 26.04.
• Run:ai 2.26: includes a read-only Run:ai MCP server that exposes cluster and workload data to AI coding assistants through the Model Context Protocol, NUMA-aware scheduling per node pool, and topology placement reporting for multi-component workloads such as NVIDIA Dynamo over Grove.
• Beta components: NVIDIA Minerva 1.0.0, described as an AIOps layer that collects network fabric telemetry Mission Control does not otherwise collect and correlates it with GPU, scheduler and workload signals; and NVIDIA Dynamic Power Software 0.9.0, which supports GB200 NVL72, GB300 NVL72 and early access Vera Rubin NVL72 but not DGX B200 or DGX B300. Both are available only through scoped engagements.
• Known limitation: energy-optimized workload power profiles are not supported on GB300 NVL72 in this release because they depend on Max-Q functionality not yet available on those systems.

Supported systems

The 2.5 release notes list component support by platform. HGX B200 and HGX B300 systems from OEMs are not supported.^[2]

The underlying hardware is covered in the articles on DGX B300, GB200 NVL72 and GB300 NVL72.

Partners

NVIDIA's product page lists Dell Technologies, Hewlett Packard Enterprise and Supermicro as system providers that have validated Mission Control on Grace Blackwell NVL72 systems.^[7] It also lists software partners whose products plug into Mission Control: Netris (network automation and multi-tenancy across Spectrum-X, Quantum InfiniBand, NVLink and BlueField), Proxmox (virtualized per-tenant control planes) and vCluster (Kubernetes tenant clusters).^[7][8] The documentation hub notes that partner software is not included with Mission Control and is supported by the partner.^[8]

Name confusion with CoreWeave Mission Control

CoreWeave sells a separate product also called Mission Control, which it describes as its "unified operating standard" for running AI workloads on CoreWeave's cloud. In December 2025 CoreWeave announced additions to it, including a Telemetry Relay, GPU straggler detection and a conversational Mission Control Agent.^[22] It is a separate product from NVIDIA's software.

Continue reading

• NvidiaNvidia Corporation is an American technology company that designs graphics processing units (GPUs) and is the world's dominant supplier of hardware for artificial intelligenceMentioned 58 times above
• NVIDIA DGXNVIDIA DGX is NVIDIA's line of integrated artificial-intelligence supercomputers: systems that package the company's highest-end data-center GPUs, CPUs, high-speed NVLink and NVSwitch interconnects…Mentioned 37 times above
• RRun:aiRun:ai (legal name Runai Labs Ltd.) is an Israeli software company that developed a Kubernetes-based orchestration and scheduling platform for graphics processing unit (GPU) resources used in artificial…Mentioned 20 times above
• KubernetesKubernetes, often abbreviated K8s, is an open-source system for deploying and managing containerized applications across a cluster of machines.Mentioned 19 times above
• SSlurmSlurm is an open-source workload manager and job scheduler for Linux clusters, developed and maintained by SchedMD, which NVIDIA acquired in December 2025.Mentioned 12 times above
• NVIDIA Vera RubinNVIDIA Vera Rubin is NVIDIA's data center AI computing platform that succeeds the NVIDIA Blackwell architecture, pairing the custom Arm-based Vera CPU with the dual-die Rubin GPU into a single superchip built…Mentioned 11 times above

More to read

• NVIDIA DGX B300
• NVIDIA DGX SuperPOD
• NVLink
• CoreWeave
• InfiniBand
• NVIDIA Minerva

Topics:AI InfrastructureData CentersMLOpsNVIDIA

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9Ryan Prout and Casey Dugas. "How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes." NVIDIA Technical Blog, 26 October 2026. developer.nvidia.com/...ous-recovery-to-kubernetes
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11NVIDIA. "NVIDIA Mission Control 2.5.0 Release Notes." NVIDIA Mission Control User's Guide. docs.nvidia.com/...nmc-release-notes
3. ^1 ^2NVIDIA AI Infrastructure (@NVIDIAAIInfra). "NVIDIA Mission Control 2.5 is now available!" X, 29 October 2026. x.com/...2115868398135763363
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7Charlie Boyle. "New NVIDIA Software for Blackwell Infrastructure Runs AI Factories at Light Speed." NVIDIA Blog, 18 March 2025. blogs.nvidia.com/...mission-control-software
5. ^1 ^2 ^3NVIDIA Newsroom. "NVIDIA Blackwell Ultra DGX SuperPOD Delivers Out-of-the-Box AI Supercomputer for Enterprises to Build AI Factories." 18 March 2025. nvidianews.nvidia.com/...upercomputer-ai-factories
6. ^1 ^2 ^3 ^4Pradyumna Desale. "Automating AI Factories with NVIDIA Mission Control." NVIDIA Technical Blog, 25 March 2025. developer.nvidia.com/...ith-nvidia-mission-control
7. ^1 ^2 ^3 ^4 ^5NVIDIA. "NVIDIA Mission Control" (product page). nvidia.com/...mission-control
8. ^1 ^2 ^3 ^4 ^5 ^6NVIDIA Docs. "NVIDIA Mission Control" (documentation hub). docs.nvidia.com/nvidia-mission-control
9. ^1 ^2NVIDIA. "Introduction: NVIDIA Mission Control Overview and Licensing." NVIDIA Base Command Manager 11 documentation. docs.nvidia.com/...introduction
10. ^NVIDIA. "NVIDIA Mission Control 1.2.1 General Availability (GA) Release" (release notes). docs.nvidia.com/...ailability-ga-release-notes.pdf
11. ^1 ^2 ^3 ^4NVIDIA. "NVIDIA Mission Control 2.0.0 Release Notes." docs.nvidia.com/...nmc-release-notes
12. ^1 ^2NVIDIA. "NVIDIA Mission Control Upgrade From 1.2 to 2.0 for GB200 NVL72 Systems." NVIDIA Mission Control Software Installation Guide 2.5.0. docs.nvidia.com/...nmc-upgrade-guide
13. ^NVIDIA. "NVIDIA Mission Control 2.1.0 Release Notes." docs.nvidia.com/...nmc-release-notes
14. ^NVIDIA. "NVIDIA Mission Control 2.2.0 Release Notes." docs.nvidia.com/...nmc-release-notes
15. ^NVIDIA. "NVIDIA Mission Control 2.3.0 Release Notes." docs.nvidia.com/...nmc-release-notes
16. ^NVIDIA. "NVIDIA Mission Control 2.3.1 Release Notes." docs.nvidia.com/...nmc-release-notes
17. ^1 ^2NVIDIA. "Introduction." NVIDIA Mission Control Software Installation Guide 2.5.0. docs.nvidia.com/...nmc-introduction
18. ^1 ^2 ^3 ^4Pradyumna Desale and Casey Dugas. "Accelerate Token Production in AI Factories Using Unified Services and Real-Time AI." NVIDIA Technical Blog, 1 April 2026. developer.nvidia.com/...-services-and-real-time-ai
19. ^Divyansh Jain. "NVIDIA's GTC 2025 Highlights: Blackwell GPUs, DGX Systems, and AI-Q Framework." StorageReview, 18 March 2025. storagereview.com/...gx-systems-and-ai-q-framework
20. ^Melania Watson. "NVIDIA launches advanced DGX SuperPOD with Blackwell GPUs." DataCentreNews UK, 19 March 2025. datacentrenews.uk/...-superpod-with-blackwell-gpus
21. ^NVIDIA. "NVIDIA Resiliency Extension (NVRx)." GitHub. github.com/...nvidia-resiliency-ext
22. ^CoreWeave. "CoreWeave Expands Mission Control To Accelerate Enterprise AI Adoption." 9 December 2025. coreweave.com/...accelerate-enterprise-ai-adoption

Show all 22 references