KAI Scheduler

KAI Scheduler is an open-source Kubernetes scheduler that optimizes the allocation of GPU resources for artificial intelligence and machine learning workloads. The scheduler originated inside the Run:ai GPU-orchestration platform and was released as a standalone open-source project by NVIDIA on April 1, 2025, under the Apache 2.0 license, following NVIDIA's December 2024 acquisition of Run:ai and the companies' pledge to open-source Run:ai's software.^[1][5][6] The project describes itself as "a robust, efficient, and scalable Kubernetes scheduler that optimizes GPU resource allocation for AI and machine learning workloads," designed to manage large GPU clusters of thousands of nodes with a high throughput of workloads.^[2] On December 21, 2025, KAI Scheduler was accepted as a sandbox project of the Cloud Native Computing Foundation (CNCF), and its repository, originally published under NVIDIA's GitHub organization, now lives in a standalone kai-scheduler GitHub organization under LF Projects, LLC governance.^[2][4]

Key facts

Scroll sideways for more →

Origin: Run:ai and the NVIDIA acquisition

Run:ai, founded by Omri Geller, Ronen Dar, and Meir Feder, built a Kubernetes-based orchestration platform that pools GPUs, splits them into fractions, and schedules AI workloads to raise the utilization of scarce accelerator hardware.^[5] NVIDIA announced a definitive agreement to acquire the Tel Aviv-based company on April 24, 2024; the companies did not disclose terms, but sources told TechCrunch the price was approximately 700 million US dollars.^[5] The acquisition closed on December 30, 2024, and Run:ai announced at closing that its software, which until then ran only on NVIDIA hardware, would be open-sourced: "Open sourcing the software will enable it to extend its availability to the entire AI ecosystem."^[6]

NVIDIA delivered on that pledge on April 1, 2025, when Run:ai co-founder Ronen Dar and Ekin Karabulut announced on the NVIDIA Technical Blog "the open-source release of the KAI Scheduler, a Kubernetes-native GPU scheduling solution, now available under the Apache 2.0 license."^[1] The post stated that the scheduler, "originally developed within the Run:ai platform," would be available to the community while continuing to be packaged and delivered as part of the commercial NVIDIA Run:ai platform, and framed the release as underscoring NVIDIA's "commitment to advancing both open-source and enterprise AI infrastructure."^[1] The project's Run:ai lineage remains visible in its API surface: the Queue custom resource definition still uses the API group scheduling.run.ai.^[7]

Problems addressed

NVIDIA's release announcement positioned KAI Scheduler against shortcomings of traditional resource schedulers when handling AI workloads on shared clusters:^[1]

• Fluctuating GPU demand. AI work shifts rapidly between small interactive jobs and multi-GPU distributed training. KAI Scheduler continuously recalculates fair-share values and adjusts quotas and limits in real time rather than relying on static assignments.^[1]
• Wait times for compute access. Combining gang scheduling, GPU sharing, and hierarchical queuing lets practitioners submit batches of jobs that launch as resources free up, in line with priorities and fairness.^[1]
• Resource guarantees and quota hogging. In shared clusters, researchers may grab more GPUs than needed early in the day to guarantee availability. The scheduler enforces per-team resource guarantees while dynamically reallocating idle capacity to other workloads.^[1]
• Tool and framework integration. A built-in "podgrouper" automatically detects and connects with workload tools such as Kubeflow, Ray, Argo, and the Training Operator, reducing manual configuration.^[1]

Architecture and scheduling cycle

KAI Scheduler is installed via Helm into its own namespace and runs alongside the default Kubernetes kube-scheduler; only pods that set schedulerName: kai-scheduler are handled by it, while all other pods follow the normal scheduling path.^[2][7] A deployment comprises several components, including an admission controller, a binder, a pod-grouper, a podgroup controller, a queue controller, and the scheduler itself.^[7] The NVIDIA GPU Operator is a prerequisite for scheduling workloads that request GPU resources.^[2]

The scheduler's two core entities are podgroups and queues.^[1] A podgroup is the atomic scheduling unit: one or more interdependent pods that must execute together as a gang, with a minimum-members count, an associated queue, and a priority class.^[1] Queues enforce resource fairness and carry four properties: a quota (the guaranteed baseline allocation), an over-quota weight (governing how surplus resources are shared), a limit (the maximum the queue can consume), and a priority.^[1]

Each scheduling cycle proceeds in three phases: taking a snapshot of cluster state, computing resource division and fair share per queue (baseline quota first, then surplus distributed in proportion to over-quota weights), and applying scheduling actions.^[1] The four actions are allocate (bind pending jobs that fit), consolidate (migrate running pods between nodes to defragment capacity for pending training jobs), reclaim (evict jobs from queues consuming more than their fair share), and preempt (displace lower-priority jobs within a queue in favor of higher-priority ones).^[1]

Features

The project's documentation lists, among others, the following capabilities:^[2]

• Batch (gang) scheduling: all pods in a group are scheduled simultaneously or not at all.
• Bin packing and spread scheduling: minimize fragmentation or, alternatively, spread load for resiliency.
• Hierarchical queues: quotas, limits, priorities, and fairness policies across multi-level queue hierarchies, for example organization-level parent queues with team-level children.
• Fairness policies: Dominant Resource Fairness (DRF) and resource reclamation across queues, plus time-based fairshare that accounts for historical usage with time decay.
• GPU sharing: multiple workloads can share single or multiple GPUs; in the fractional model a pod requests a slice via a gpu-fraction annotation.^[2][7]
• Topology-Aware Scheduling (TAS) and hierarchical PodGroups for placing multi-level distributed and disaggregated workloads.
• Dynamic Resource Allocation (DRA): vendor-specific hardware through Kubernetes ResourceClaims, covering GPUs from NVIDIA or AMD, including NVIDIA ComputeResources on GB200/GB300 systems.
• Elastic workloads, workload consolidation, minimum-guaranteed runtime before preemption, and workload priority decoupled from preemptibility.

The README notes that KAI Scheduler is built on kube-batch, the earlier Kubernetes batch-scheduling project, and supports both cloud (including autoscalers such as Karpenter) and on-premises deployments.^[2] For GPU computing workloads that share a device, the scheduler decides placement and proportions but does not enforce hardware-level memory isolation; the GPU time-slices between the CUDA contexts of the sharing pods, so applications must respect their assigned memory budgets, while NVIDIA Multi-Instance GPU (MIG) partitions, which KAI Scheduler can also schedule, provide hard isolation on supported hardware.^[7]

Releases, governance, and adoption

The GitHub repository was created in late February 2025, ahead of the April 1, 2025 public announcement.^[1][2] Version 0.10.0, released in November 2025, added Topology-Aware Scheduling, hierarchical PodGroups, and time-based fairshare.^[2] Release cadence has been rapid: six v0.16.x point releases shipped in July 2026 alone, and v0.17.0 was published on August 3, 2026.^[3] The project runs public dashboards for daily scale tests that validate scheduler performance at cluster sizes from hundreds to thousands of nodes, alongside per-commit performance benchmarks.^[2]

After the CNCF accepted KAI Scheduler as a sandbox project on December 21, 2025, the project adopted community infrastructure including a #kai-scheduler channel on the CNCF Slack, a public mailing list, and bi-weekly community calls.^[2][4] Ecosystem integrations documented by the project include native support for Ray workloads through KubeRay (October 2025) and integration of its topology-aware and hierarchical gang scheduling with NVIDIA's Grove and Dynamo projects for disaggregated inference serving (November 2025).^[2] Project talks were presented at KubeCon North America 2025, on topology-aware scheduling, and KubeCon Europe 2026, on GPU reservations across teams.^[2]

Multi-tenant GPU sharing with vCluster

On August 3, 2026, the NVIDIA Technical Blog published a tutorial by Ekin Karabulut (NVIDIA, previously at Run:ai) and Saiyam Pathak (head of developer relations at vCluster) demonstrating a multi-tenancy pattern that combines KAI Scheduler with vCluster, the open-source virtual Kubernetes cluster tool from Loft Labs.^[7] The pattern targets organizations where teams want their own clusters for autonomy over CRD versions, RBAC, and cluster-admin access, but where dedicating physical GPU hardware per team would strand capacity.^[7]

In the walkthrough, three teams (NLP, computer vision, and recommender systems) each receive a fully isolated virtual Kubernetes cluster, with its own API server, CRDs, and RBAC, while all three share one NVIDIA L40S GPU (48 GB of VRAM) on a single node.^[7] A KAI Scheduler queue hierarchy models the organization: a parent queue holds a budget of one GPU, and each team's child queue is guaranteed a 0.33 GPU quota with the ability to burst to the full GPU when the others are idle.^[7] Each team's pod requests its slice through the gpu-fraction annotation, and all three pods run concurrently on the same physical GPU while each team sees only its own workloads from inside its virtual cluster.^[7] The tutorial notes one integration detail: vCluster's owner-rewriting must be disabled (setOwner: false) so that KAI Scheduler's pod-grouper can walk real ownership chains, such as Job to Pod, to group workloads automatically.^[7]

NVIDIA promoted the tutorial through its NVIDIA AI Infrastructure account on X, writing that "the answer isn't always more GPUs, it's better utilization of the ones you have," and describing three teams running isolated workloads on a single L40S "with zero wasted capacity."^[8] The blog post states that the same pattern scales to clusters with hundreds of GPU nodes and dozens of teams, and that a KAI Scheduler and vCluster integration session was planned for KubeCon North America 2026, November 9 to 12.^[7]

See also

• Run:ai
• NVIDIA
• GPU
• GPU cluster
• GPU computing

References

1. ^Ronen Dar and Ekin Karabulut. "NVIDIA Open Sources Run:ai Scheduler to Foster Community Collaboration." NVIDIA Technical Blog. April 1, 2025. developer.nvidia.com/...er-community-collaboration
2. ^KAI Scheduler project. "KAI-Scheduler repository README." GitHub. Retrieved August 5, 2026. github.com/...KAI-Scheduler
3. ^KAI Scheduler project. "Releases." GitHub. Retrieved August 5, 2026. github.com/...releases
4. ^Cloud Native Computing Foundation. "KAI Scheduler." CNCF project listing. Retrieved August 5, 2026. cncf.io/...kai-scheduler
5. ^TechCrunch. "Nvidia acquires AI workload management startup Run:ai for $700M, sources say." April 24, 2024. techcrunch.com/...orkload-management-startup-runai
6. ^TechCrunch. "Nvidia completes acquisition of AI infrastructure startup Run:ai." December 30, 2024. techcrunch.com/...-ai-infrastructure-startup-runai
7. ^Ekin Karabulut and Saiyam Pathak. "How to Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure." NVIDIA Technical Blog. August 3, 2026. developer.nvidia.com/...-shared-gpu-infrastructure
8. ^NVIDIA AI Infrastructure (@NVIDIAAIInfra). Post on X. August 3, 2026. x.com/...2084377877630332963