Before discussing advanced Slurm usage, it is important to take a step back and understand the fundamental principles at play during the lifecycle of a job managed by Slurm. In particular, we shall focus on three areas:

Allocation
How, when, and why Slurm allocates resources for a job.

Execution
What Slurm runs on allocated resources, and how.

Security
What access is allowed to allocated resources, and how this is achieved.

Slurm features a highly extensible infrastructure that can be configured to allow diverse modes of operation; as such we will not cover all areas comprehensively. We will focus on common scenarios in large-scale HPC configurations, neglecting other types of Slurm configurations that tend to increase the chances that users' jobs will interfere with each other, or overburden the scheduler. Therefore, this topic focuses just on the Slurm configuration at TACC, as its HPC systems are reasonably representative of large-scale systems at other HPC centers. This means we assume a Slurm setup with the following characteristics:

• Job allocations may not occupy nodes partially, or split the nodes' resources with other jobs.
• Jobs may not define "job arrays" that rely on Slurm's scheduler to manage tasks within a job.
• The use of Slurm's srun command to execute "job steps" within a job is not supported.
• For MPI jobs, Slurm's plugin to the srun command is not used to launch MPI tasks.

While the MPI plugin mentioned above is utilized in some HPC systems such as Expanse at SDSC, TACC provides its own, more feature-rich MPI launcher, ibrun, which works very well with Slurm.