Created by Frank Schluenzen, last modified on Oct 14, 2020 15:40

Setting warpx environment

warpx modules do not provide a real warpx-binary, but only set the environment enabling warpx compilation!


@max-wgse001:~$ module load maxwell warpx
 
 Loading modules: gcc/9.3 openmpi/4.0.4 hdf5/1.10.6 
 Adios2:          /software/gcc/9.3.0/openmpi/4.0.4/adios2/2.6.0 
 openPMD:         /software/gcc/9.3.0/openmpi/4.0.4/openPMD-api/0.12.0
 warpX:           /software/gcc/9.3.0/openmpi/4.0.4/warpx_directory - sample installation

Additional variables defined:
 ADIOS2:           /software/gcc/9.3.0/openmpi/4.0.4/adios2/2.6.0
 openPMD:          /software/gcc/9.3.0/openmpi/4.0.4/openPMD-api/0.12.0

Building warpx:
 should work out of the box: module load naxwell warpx; make USE_MPI=TRUE USE_OPENPMD=TRUE 
 with GPU support: module load naxwell warpx cuda/11.0; make USE_MPI=TRUE USE_OPENPMD=TRUE USE_GPU=TRUE 
 openpmd backends supported: bp,h5 (adios2,hdf5) 

# On nodes equipped with a GPU you could in addition enable CUDA
@max-wgse001:~$ module load cuda/11.0

Building warpx

# environment
module load maxwell warpx
module load cuda/11.0

# get the code and dependencies
cd /scratch/me/
rm -rf warpx_directory/
mkdir warpx_directory
cd warpx_directory/
git clone https://github.com/ECP-WarpX/WarpX.git
git clone https://github.com/ECP-WarpX/picsar.git
git clone --branch development https://github.com/AMReX-Codes/amrex.git
cd WarpX

# build in place
make USE_MPI=TRUE USE_OPENPMD=TRUE USE_GPU=TRUE  # USE_GPU=FALSE on non-gpu nodes
# worked without errors both for CPU and GPU modes

Testing the backends

# get sample input file
cp /software/gcc/9.3.0/openmpi/4.0.4/warpx_directory/WarpX/Bin/inputs_3d .

# test the hdf5 backend
perl -pi -e 's|diag1.openpmd_backend =.*|diag1.openpmd_backend = h5|' inputs_3d
mpirun -np 4 /scratch/warpx_directory/me/WarpX/Bin/main3d.gnu.TPROF.MPI.CUDA.OPMD.ex inputs_3d

# test the bp backend 
perl -pi -e 's|diag1.openpmd_backend =.*|diag1.openpmd_backend = bp|' inputs_3d
mpirun -np 4 /scratch/warpx_directory/me/WarpX/Bin/main3d.gnu.TPROF.MPI.CUDA.OPMD.ex inputs_3d

# output generated
ls -lrt diags/diag1/
-rw-r--r-- 1 me it 169496112 Sep 15 12:14 openpmd_000000.h5
-rw-r--r-- 1 me it 225484720 Sep 15 12:14 openpmd_001000.h5
drwxr-xr-x 2 me it 4096 Sep 15 12:23 openpmd_000000.bp
drwxr-xr-x 2 me it 4096 Sep 15 12:23 openpmd_001000.bp