In addition to the standard gromacs installation, there are customized versions available on Maxwell. This is currently still under construction, so you might notice minor changes from time to time. Modulefiles are currently also missing, but will be provided soon.
Running gromacs
gromacs is compiled using gcc/8.2 and - for the openmpi version - with openmpi/4.0.3. That is currently the only combination which works properly on recent hardware with Mellanox-ConnectX6 infiniband adapters. A job script should look like this:
#!/bin/bash
#SBATCH --partition=maxcpu
#SBATCH --time=0-16:00:00
#SBATCH --job-name=gromacs
unset LD_PRELOAD
source /etc/profile.d/modules.sh
# the required setup. Note: the modules have to be loaded in exactly this order!
module load maxwell gcc/8.2 openmpi/4.0.3
export PATH=/software/gromacs/2020.2/bin:$PATH
# ------------------------------------------------------------------------------------
# everything else is identical for the standard gromacs:
base=/beegfs/desy/user/$USER/GROMACS
project=20k
rm -rf $base/$project
mkdir -p $base/$project
pushd $base/$project
# fetch the benchmark sample:
cp /beegfs/desy/group/it/Benchmarks/gromacs/HECBioSim/${project}-atoms/benchmark.tpr .
# gromacs doesn't benefit form hyperthreaded cores, so just use phyical cores.
# gromacs recommends between 1 and 6 threads per mpi rank, so as an example we use 2 threads per rank
# which means we have mpi-ranks= "total number of cores" / (2*2)
total_cores=$(scontrol show job $SLURM_JOB_ID | grep NumCPUs | cut -d= -f3 | awk '{print $1}')
np=$(( $total_cores / 4 ))
nt=2
INPUT=benchmark.tpr
OUTPUT=benchmark.log
STARTTIME=$(date +%s)
SLURM_JOB_NUM_NODES
# the actual gromacs run
# just use as many MPI processes as nodes. For each node use nt threads:
mpirun --map-by node -np $np `which mdrun_openmpi` -ntomp $nt -s ${INPUT} -g ${OUTPUT}
ENDTIME=$(date +%s)
ELAPSED=$(($ENDTIME - $STARTTIME))
x=$(grep Performance $OUTPUT)
cat <<EOF
Time elapsed: $ELAPSED seconds
NodeList: $SLURM_JOB_NODELIST
Number of nodes: $SLURM_JOB_NUM_NODES
Number of mpi ranks: $np
Number of threads/rank: $nt
(ns/day) (hour/ns)
$x
EOF
popd
exit
################################################################
# output:
Time elapsed: 101
NodeList: max-wn[012,061]
Number of nodes: 2
Number of mpi ranks: 20
Number of threads/rank: 2
(ns/day) (hour/ns)
Performance: 86.329 0.278
Building gromacs
the instructions used to build gromacs:
cd /scratch/$USER
tar xf /software/gromacs/src/gromacs-2020.2.tar.gz
cd gromacs-2020.2
prefix=/software/gromacs/2020.2
module purge
module load maxwell gcc/8.2
rm -rf build; mkdir build; cd build
CC=/software/gcc/8.2/bin/gcc cmake .. -DCMAKE_INSTALL_PREFIX:PATH=$prefix -DGMX_MPI=off -DGMX_BUILD_OWN_FFTW=ON -DGMX_SIMD=AVX2_128
make -j 8
make install
#
cd ..
rm -rf build; mkdir build; cd build
module purge
module load maxwell gcc/8.2 openmpi/4.0.3
cmake .. -DCMAKE_C_COMPILER=mpicc \
-DCMAKE_CXX_COMPILER=mpicxx \
-DGMX_MPI=ON -DCMAKE_INSTALL_PREFIX:PATH=$prefix -DGMX_MPI=on -DGMX_OPENMP=on \
-DGMX_BUILD_MDRUN_ONLY=ON -DGMX_HWLOC=on -DGMX_BUILD_OWN_FFTW=ON -DGMX_SIMD=AVX2_128
make -j 8
make install
Please note: the selection of GMX_SIMD=AVX2_128 will change later on.