Slurm

Slurm is an open-source workload manager and job scheduler for Linux clusters, developed and maintained by SchedMD, which NVIDIA acquired in December 2025 ^[9][10]. The software decides which jobs run where and when on a shared cluster: it allocates nodes to users for some period of time, provides a framework for starting and monitoring parallel work on those nodes, and arbitrates contention through a queue of pending jobs ^[1]. Slurm is distributed under version 2 of the GNU General Public License and has drawn contributions from more than 250 people since its first release in 2002 ^[4].

Originally an acronym for Simple Linux Utility for Resource Management (the name also nods to the soda in Futurama), Slurm grew from a Lawrence Livermore National Laboratory project into the dominant scheduler in high-performance computing ^[3][4]. As of November 2021 it ran on about 60 percent of the TOP500 supercomputers, and NVIDIA stated at the time of the SchedMD acquisition that Slurm is used on more than half of the top 10 and top 100 TOP500 systems ^[4][10][12]. The same qualities that made it standard in scientific computing, gang scheduling of tightly coupled parallel jobs, fair-share accounting, and topology-aware placement, later made it the default batch system on many of the GPU clusters used to train large language models ^[10][12].

Slurm's position in AI infrastructure has pulled it into the long-running comparison with Kubernetes, the container orchestrator that grew up in cloud data centers. SchedMD's answer is Slinky, a set of open-source projects that run Slurm inside Kubernetes clusters or let Slurm schedule Kubernetes workloads, and GPU cloud providers such as CoreWeave ship their own Slurm-on-Kubernetes stacks ^[13][14][15].

History

Slurm began as a collaborative development effort involving Lawrence Livermore National Laboratory (LLNL), Linux NetworX, Hewlett-Packard, and Groupe Bull, with the first release in 2002 ^[4]. The design borrowed from the closed-source Quadrics RMS scheduler and kept a similar command syntax ^[4]. The system was described publicly in a 2003 paper, "SLURM: Simple Linux Utility for Resource Management," by Andy Yoo, Morris Jette, and Mark Grondona, published in the Job Scheduling Strategies for Parallel Processing proceedings (Lecture Notes in Computer Science volume 2862) ^[3].

In 2010, lead developers Morris Jette and Danny Auble founded SchedMD LLC to develop Slurm and sell commercial support ^[4][11]. SchedMD maintains the canonical source tree; the "Simple Linux Utility" acronym was eventually dropped in favor of the plain name Slurm Workload Manager ^[4]. Along the way the scheduler was ported across most major supercomputing architectures, including IBM BlueGene/Q machines such as the 20-petaflop Sequoia and Cray XT, XE, and Cascade systems, as well as China's Tianhe-2, which had 3.1 million cores ^[4].

On December 15, 2025, NVIDIA announced it had acquired SchedMD for an undisclosed sum ^[10][11]. NVIDIA, which had worked with SchedMD for over a decade, committed to continuing development and distribution of Slurm as open-source, vendor-neutral software, and SchedMD CEO Danny Auble said the software would remain open source ^[10]. NVIDIA framed the purchase around AI workloads, describing Slurm as the leading workload manager in scalability, throughput, and complex policy management, with SchedMD serving hundreds of customers across cloud providers, manufacturers, AI companies, and research labs in industries from autonomous driving to financial services ^[10].

Architecture

Slurm is built around a small set of daemons plus a plugin mechanism. A central controller, slurmctld, monitors resources and work, optionally with a backup instance for failover. Each compute node runs slurmd, which the documentation compares to a remote shell: it waits for work, executes it, returns status, and waits for more ^[1]. An optional slurmdbd (Slurm DataBase Daemon) records accounting information for multiple clusters in a single database, and slurmrestd exposes a REST API ^[1]. Slurm requires no kernel modifications and is described by its developers as fault-tolerant and highly scalable; Wikipedia's summary credits it with scheduling up to 100,000 independent jobs on the 100,000 sockets of IBM Sequoia and sustaining up to 1,000 job submissions per second ^[1][4].

The entities Slurm manages are nodes (the compute resources), partitions (logical groupings of nodes that act as job queues, each with its own limits on size, time, and users), jobs (allocations of resources to a user for a period of time), and job steps (sets of possibly parallel tasks within a job's allocation) ^[1][2]. A single job can contain many job steps, which carries less overhead than submitting each piece of work as its own job ^[2].

The main user commands:

Scroll sideways for more →

Nearly everything else is a plugin: authentication, accounting, energy monitoring, MPI support, container support, preemption, scheduling policy, network topology, task affinity, and generic resources all load as interchangeable modules ^[1].

Scheduling features

Job priority under the widely used multifactor priority plugin is a weighted sum of factors including job age (time spent waiting), fair-share (how much of its allocation an account has already consumed), job size, partition priority, Quality of Service (QOS), per-resource TRES factors, and a user-adjustable nice value ^[8]. Weights are 32-bit integers set by administrators, so sites can make fair-share dominate, favor large jobs to drain space for full-machine runs, or favor short jobs, depending on policy ^[8].

Preemption lets high-priority work displace running jobs. With PreemptType=preempt/partition_prio, jobs in higher-priority partitions can preempt jobs in lower-priority ones; with preempt/qos, the preemption rules come from QOS definitions in the Slurm database ^[7]. PreemptMode controls what happens to the victim: CANCEL kills it, REQUEUE puts it back in the queue, and SUSPEND pauses it so the gang scheduler can resume it after the preemptor finishes; a GraceTime parameter gives preempted jobs a window to shut down cleanly between SIGTERM and SIGKILL ^[7]. The same gang scheduling machinery can time-slice jobs that oversubscribe the same resources ^[7].

Job arrays handle collections of similar jobs. A single sbatch --array=0-31 submission creates 32 tasks; syntax variants allow explicit index lists (--array=1,3,5,7), strides (--array=1-7:2), and a throttle on concurrency (--array=0-15%4 keeps at most four tasks running at once) ^[6]. Each task sees its index in the SLURM_ARRAY_TASK_ID environment variable. The maximum index is bounded by the MaxArraySize parameter, which defaults to 1001 and can be raised to 4,000,001, and the documentation notes that arrays with millions of tasks can be submitted in milliseconds ^[6].

GPU scheduling

GPUs enter Slurm through the generic resource (GRES) mechanism, which handles arbitrary consumable resources attached to nodes ^[5]. Administrators declare GresTypes=gpu in slurm.conf and describe each node's devices either manually in gres.conf (device file, type such as a100, and the CPU cores with good affinity to it) or automatically: AutoDetect=nvml uses the NVIDIA Management Library and picks up MIG instances and NVLink topology, rsmi covers AMD GPUs through ROCm, and oneapi covers Intel GPUs ^[5].

Users request GPUs with --gres=gpu:2 or the newer per-job options --gpus, --gpus-per-node, --gpus-per-task, and --gpus-per-socket, plus companions such as --cpus-per-gpu, --mem-per-gpu, --gpu-bind, and --gpu-freq ^[5]. Slurm sets CUDA_VISIBLE_DEVICES for each job step so processes only address their allocated devices, and with cgroup enforcement the unallocated device files are not visible to the job at all ^[5]. Three sharing modes exist for carving up a physical GPU: NVIDIA Multi-Process Service (MPS) allocates percentage slices of one GPU across jobs, Multi-Instance GPU (MIG) partitions supported NVIDIA GPUs into isolated instances that Slurm treats as separate GPUs, and shards provide simple GPU sharing without process isolation ^[5]. Accounting can track GPU allocation, GPU memory, and GPU utilization as trackable resources (gres/gpu, gres/gpumem, gres/gpuutil) ^[5].

This machinery, along with topology-aware placement of multi-node jobs, is the core of Slurm's role in AI training, where a single distributed training job may hold thousands of GPUs for weeks and any scheduling inefficiency is expensive ^[10][12].

Adoption in HPC and AI

Slurm is the batch system on a large share of the world's biggest machines. Frontier, the exascale system at Oak Ridge National Laboratory with 9,856 AMD compute nodes, schedules work through Slurm, which treats each of the eight GPU compute dies per node as a separate GPU ^[16]. The EuroHPC LUMI system uses Slurm as its job scheduler and resource manager ^[17], as do many national and university clusters. In the cloud, Amazon Web Services ParallelCluster deploys HPC clusters with Slurm as a supported scheduler ^[19], and NVIDIA's Mission Control cluster software supports both Slurm and Kubernetes for multi-node scheduling ^[12].

AI training runs have depended on it too. The BLOOM 176-billion-parameter model was trained in 2022 on the French Jean Zay supercomputer using SLURM to manage 384 NVIDIA A100 80GB GPUs across 48 nodes with the Megatron-DeepSpeed framework ^[18]. The BLOOM team's writeup also recorded a practical complaint: "SLURM wasn't designed to be used by a team of people. A SLURM job is owned by a single user and if they aren't around, the other members of the group can't do anything to the running job," which led them to build a kill-switch workaround so other group members could kill a running process when its owner was away ^[18]. GPU cloud providers aimed at AI labs, including CoreWeave, ship managed Slurm environments so training teams can keep their existing batch workflows ^[15].

Slurm and Kubernetes

Slurm and Kubernetes approach cluster management from opposite ends. Slurm assumes batch jobs with fixed lifetimes, gang-scheduled across many nodes, submitted by users who wait in a queue; Kubernetes assumes long-running containerized services that scale up and down. AI organizations often need both models on the same hardware: training looks like HPC batch work, while inference serving looks like a Kubernetes workload. OpenAI took the pure-Kubernetes route early, reporting that it scaled Kubernetes clusters to 7,500 nodes to produce infrastructure for large models such as GPT-3, CLIP, and DALL-E ^[21].

SchedMD's bridge is Slinky, described as its set of projects to enable interoperability between Slurm and Kubernetes ^[13]. The main components are slurm-operator, a Kubernetes operator that runs and scales Slurm clusters as pods (with NodeSets for worker nodes and support for hybrid deployments where some Slurm components live outside Kubernetes), slurm-bridge, which lets Slurm act as a scheduler for Kubernetes workloads, a Go client library for the Slurm REST API, and reference container images ^[13][14]. The slurm-operator is Apache 2.0 licensed, has shipped stable v1.x releases, and requires at least Kubernetes 1.29 and Slurm 25.11 ^[14].

CoreWeave built an independent equivalent called SUNK (Slurm on Kubernetes), which runs each Slurm node as a Kubernetes pod containing a slurmd container so that training jobs under Slurm and inference services under Kubernetes share one cluster ^[15]. Deployment is through a Helm chart covering the controller, accounting, and topology configuration, with base images that preinstall CUDA, NCCL, and InfiniBand libraries ^[15].

Limitations and alternatives

Slurm's single-user job ownership model, noted above in the BLOOM case, fits poorly with team-operated training runs ^[18]. Its configuration surface is large, and the multi-tenant, service-oriented features that Kubernetes provides natively require extra tooling on a Slurm cluster; that gap is what Slinky and SUNK exist to fill ^[13][15].

Alternatives persist at both ends of the spectrum. Kubernetes itself, with batch add-ons, is the main competitor in AI-native shops ^[21]. In traditional HPC, LLNL, where Slurm originated, developed the Flux resource management framework as a designed-from-scratch replacement; the Flux documentation notes that Flux "was designed from the beginning to replace Slurm" but "did not gain momentum as a system workload manager until 2024 with the early deliveries of El Capitan," LLNL's exascale system, which Flux now manages ^[20]. Other established schedulers in the same niche include PBS Professional, developed by Altair and now sold by Siemens ^[24], and IBM's LSF ^[4], though neither matches Slurm's TOP500 footprint ^[4][10].

The NVIDIA acquisition added a new question: whether a scheduler that must stay neutral across AMD, Intel, and NVIDIA hardware can be owned by one GPU vendor. NVIDIA's answer so far has been explicit public commitments to keep Slurm open source and vendor-neutral ^[10].

Recent developments

Slurm switched to a year.month version scheme, and releases have arrived on a roughly six-month cadence. Slurm 25.11, the version the Slinky operator targets, shipped in late 2025 and continues to receive maintenance releases (25.11.7 arrived July 14, 2026) ^[14][22]. Slurm 26.05.0 was released on May 26, 2026, followed by 26.05.1 in June and 26.05.2 on July 14, 2026 ^[22].

The 26.05 release notes lean toward large GPU clusters and observability: an asynchronous srun step mode that queues step processes through the step manager, new ring and 3D torus topology plugins, the ability for a running job to release memory, with its cgroup limits updated accordingly, expanded REST API coverage, a faster single-node scheduling path, and Prometheus/OpenMetrics endpoints exposing GPU allocation statistics across nodes, partitions, and jobs ^[23].

Since the acquisition closed, SchedMD has operated as part of NVIDIA while keeping its commercial support, training, and deployment services under the SchedMD name, and continues to describe Slurm as the market-leading open-source workload manager for HPC and AI infrastructure ^[9].

See also

• AI infrastructure
• GPU cluster
• Distributed training
• CoreWeave
• NVIDIA
• CUDA

References

1. ^SchedMD, "Slurm Workload Manager: Overview," Slurm documentation. slurm.schedmd.com/overview
2. ^SchedMD, "Quick Start User Guide," Slurm documentation. slurm.schedmd.com/quickstart
3. ^SchedMD, "Publications and Presentations," Slurm documentation (lists A. Yoo, M. Jette, and M. Grondona, "Slurm: Simple Linux Utility for Resource Management," Job Scheduling Strategies for Parallel Processing, LNCS vol. 2862, Springer, 2003). slurm.schedmd.com/publications
4. ^Wikipedia, "Slurm Workload Manager." en.wikipedia.org/...Slurm_Workload_Manager
5. ^SchedMD, "Generic Resource (GRES) Scheduling," Slurm documentation. slurm.schedmd.com/gres
6. ^SchedMD, "Job Array Support," Slurm documentation. slurm.schedmd.com/job_array
7. ^SchedMD, "Preemption," Slurm documentation. slurm.schedmd.com/preempt
8. ^SchedMD, "Multifactor Priority Plugin," Slurm documentation. slurm.schedmd.com/priority_multifactor
9. ^SchedMD, company homepage. schedmd.com
10. ^NVIDIA, "NVIDIA Acquires SchedMD," NVIDIA Blog, December 15, 2025. blogs.nvidia.com/...nvidia-acquires-schedmd
11. ^TechCrunch, "Nvidia bulks up open source offerings with an acquisition and new open AI models," December 15, 2025. techcrunch.com/...quisition-and-new-open-ai-models
12. ^NVIDIA, "Slurm Workload Manager," nvidia.com software page. nvidia.com/...slurm
13. ^SchedMD, "Slinky," Slurm documentation. slurm.schedmd.com/slinky
14. ^SlinkyProject, "slurm-operator: Run Slurm on Kubernetes," GitHub repository. github.com/...slurm-operator
15. ^CoreWeave, "SUNK (Slurm on Kubernetes)," CoreWeave documentation. docs.coreweave.com/...sunk
16. ^Oak Ridge Leadership Computing Facility, "Frontier User Guide." docs.olcf.ornl.gov/...frontier_user_guide
17. ^LUMI consortium, "Slurm quickstart," LUMI documentation. docs.lumi-supercomputer.eu/...slurm-quickstart
18. ^Stas Bekman, "The Technology Behind BLOOM Training," Hugging Face Blog, 2022. huggingface.co/...bloom-megatron-deepspeed
19. ^Amazon Web Services, "What is AWS ParallelCluster?," AWS ParallelCluster User Guide. docs.aws.amazon.com/...what-is-aws-parallelcluster
20. ^Flux Framework, "Migrating from Slurm," flux-core administration guide. flux-framework.readthedocs.io/...admin_slurm
21. ^OpenAI, "Scaling Kubernetes to 7,500 nodes," OpenAI blog. openai.com/...scaling-kubernetes-to-7500-nodes
22. ^SchedMD, Slurm release tags, GitHub. github.com/...tags
23. ^SchedMD, "Release Notes for Slurm 26.05," Slurm documentation. slurm.schedmd.com/release_notes
24. ^Siemens, "PBS Professional," HPCWorks product page. siemens.com/...pbs-professional