Loughborough University
Leicestershire, UK
LE11 3TU
+44 (0)1509 222222
Loughborough University

IT Services : High Performance Computing

Running Mixed Parallel Jobs


Mixed mode MPI and OpenMP

Here MPI is used to communicate between nodes, and OpenMP is used to provide multicore parallelism on a single node. In this case you should set the CPUs per task to be the number of cores on a node and then set the number you need. Setting OMP_NUM_THREADS is as for an Running OpenMP Jobs program as are the issues with partitions.

#!/bin/bash
#SBATCH --time=00:1:00
#SBATCH --job-name=openmpstuff
#SBATCH --output=slurm.out
#SBATCH --error=slurm.err
#SBATCH --partition=compute
#SBATCH --nodes=2
#SBATCH --cpus-per-task=12
#SBATCH --account=ITTEST
#SBATCH --mail-type=ALL
#SBATCH --mail-user=a.turner@lboro.ac.uk

export OMP_NUM_THREADS=$SLURM_JOB_CPUS_PER_NODE

# Assuming bullxmpi was used during compilation...
module load bullxmpi

mpirun program.exe 
      

Running when the number of cores required is less than a node

Use an allocation (see Allocations ) and submit with sbatch into this. The number of --cpus-per-task will be set by the salloc command which requires the number of threads for OpenMP portion to be calculated differently

More advice and examples will be added in due course.