Maxwell : qchem

Q-Chem, Inc. is a comprehensive ab initio quantum chemistry program to solve computational problems faster, more accurately and less expensively than ever before possible. Q-Chem's capabilities facilitate applications in pharmaceuticals, materials science, biochemistry and other fields.  Q-Chem also provides users with the highest level of technical support. Q-Chem was established in 1993, and there have been five major releases to date. (from: https://www.q-chem.com/about/)

Using  qchem

qchem is available in two variants, as a shared memory (shm) and as a cluster/mpi version mit openmpi. The environment can best be initiated using the module command:

[max]% module load maxwell qchem/shm                 # for the shared memory version. Don't see a need to use it at all.
[max]% module load maxwell qchem/mpi                 # for cluster/mpi enabled version
[max]% module load maxwell qchem                     # for cluster/mpi enabled version, which is the default. 

Note: the modules define QCSCRATCH as

• /scratch/$USER/qcscratch - for the shm version
• /beegfs/desy/user/$USER/qcscratch - for the mpi version

Alter the environment variable if you favor a different location. Make sure /beegfs/desy/user/$USER exists before launching jobs (mk-beegfs will do).

Note: the mpi module uses mpi/mpich-3.2-x86_64. If you favor another MPI implementation use e.g.

[max]% module load maxwell qchem        
[max]% module unload mpi/mpich-3.2-x86_64 
[max]% module load <your favorite MPI>

Sample job script

qchem setup is a bit tricky. Here is a sample script which basically works. Note that there is hardly an

#!/bin/bash
#SBATCH --time=0-08:00:00
#SBATCH --partition=upex     # change!
#SBATCH --constraint=7542    # change!
#SBATCH --nodes=4            # change!
unset LD_PRELOAD
#
#  qchem setup
#
source /etc/profile.d/modules.sh
module purge
module load maxwell qchem/mpi

#
#  use sample for naming files
#
sample=N2O-mod

#
#  use one mpi-process per node
#  use number of physical cores as number of threads, assuming identical core count across nodes
#
nc=$(( $(nproc) / 2 ))
np=$SLURM_NNODES
nt=$nc

#
#  generate a hostfile. This one uses 1 MPI process per node (:1)
# 
sinfo --noheader -n $SLURM_JOB_NODELIST  -o "%n:1"  > hostfile
export QCMACHINEFILE=$PWD/hostfile

#
#  use 75% of available memory (on the node with lowest memory). Allow 50G for other applications (i.e. gpfs)
#  add memory setting to input file
#
mem=$( scontrol show node $SLURM_JOB_NODELIST | grep -o 'mem=.*G' | cut -d= -f2 | tr -d G | sort -u | head -1)
mem=$(( ($mem - 50) * 750 ))
perl -p -e "s|MEM_TOTAL.*|MEM_TOTAL $mem|" $sample.tmpl > $sample.in

#
#  just to have some info in the slurm-log as well
#
echo "$(date) sample: $sample threads: $nt mpi: $np mem_total: $mem "

s=$(date +%s)
qchem -mpi -np $np -nt $nt $sample.in $sample.out > $sample.log 2>&1
e=$(date +%s)
elapsed=$(( $e - $s ))

#
#  all done 
#
echo "$(date) sample: $sample threads: $nt mpi: $np elapsed: $elapsed "

exit

Setting up partitions and constraints

In general you want to use a set of fairly identical nodes for MPI jobs. To find a suitable set of constraints:

• use my-partitions to determine which partition to use. Don't use the jhub partition for jobs!
• use something like sinfo -p <favorite partitions> -o '%n %f %t' to determine what kind of constraints are available in which partition
• use savail -f <constraint> to show real availability of nodes

Test run

Some test runs for 1-8 nodes using the mpi-version show, that with the above job-script (which might not be appropriate) the use of multiple nodes is barely improving performance - if at all. That might well be due to the size of the problem. The jobs however make intense use of filesystems. Neither BeeGFS nor GPFS are particularly happy with that. The jobs running on GPFS all failed exceeding the time limit of 8 hours. Some jobs using BeeGFS didn't terminate properly unable to close files. Using the INCORE backend, the jobs ran - seemingly smoothly and almost a factor 10 faster.

The resulting wall clock times