In this exercise we will try submitting batch and interactive jobs through Slurm. It is assumed you will want to use the ompsumtest.c and ompipi.c codes from the exercise on compiling, but feel free to pick your own OpenMP and MPI program(s) to compile and run if you wish. Please refer to the Compiling topic if you want a more thorough introduction to the compilers.

First, make sure there are nodes available in the development queue by issuing

$ sinfo -S+P -o "%18P %8a %20F"

If you haven't done so already, compile the OpenMP program ompsumtest.c with icc:

$ icc ompsumtest.c -O3 -xHost -qopenmp -o ompsumtest

Note, the OpenMP program is so tiny that the optimization options don't really matter, but it's good practice to include them anyway. Now let's define a batch job that will execute this code on a development node. Create the following script in an editor (or with cat >) and call it myJob.sh:

#!/bin/bash
#SBATCH -p development
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 00:10:00
# Uncomment next line and specify a project, if you have more than 1 project
##SBATCH -A <your account>

export OMP_NUM_THREADS=56
./ompsumtest

Submit your batch script with sbatch:

$ sbatch myJob.sh

An output file will appear when the job starts running. View it to verify that the job ran correctly:

$ cat slurm-<job number>.out

If something went wrong, then check the error file instead, and try again.

Next, let's do some interactive experimentation with the other code, ompipi.c. This is a hybrid OpenMP/MPI program, so it will be interesting to explore the effects of choosing different numbers of threads and processes (where each MPI task is a process). Compile the code as follows:

$ mpicc ompipi.c -O3 -xHost -qopenmp -o ompipi

Start up an interactive shell on a development node:

$ idev

When the prompt appears, run ompipi with a very small number of threads and tasks, like this:

$ export OMP_NUM_THREADS=1
$ ibrun -n 1 ./ompipi 10000000

Gradually increase the number of threads and/or tasks until the product (threads) × (tasks) is 56, so that all cores are occupied. (The code doesn't scale well to 56 because there is not enough work to do; ompipi requires a large argument so the computation runs in a measurable amount of time.) You can also play with the argument to ompipi to see what effect it has on the precision of the answer and the timing. When you're finished, type ctrl-D to exit the shell and terminate the job.