Created by Frank Schluenzen, last modified on Oct 14, 2020 17:17

FBPIC is a Particle-In-Cell (PIC) code for relativistic plasma physics. It is especially well-suited for physical simulations of laser-wakefield acceleration and plasma-wakefield acceleration, with close-to-cylindrical symmetry.

Installing fbpic in virtual environment

module purge
cd /software/fbpic
python3 -m venv 0.18
source 0.18/bin/activate
export LLVM_CONFIG=/usr/lib64/llvm4.0/bin/llvm-config
pip install --upgrade pip
pip install --upgrade numpy scipy pyfftw cupy-cuda102 llvmlite
pip install fbpic
deactivate

Running fbpic interactively

# where my samples reside
cd /beegfs/desy/user/schluenz/fbpic

# get two GPU nodes
salloc -N 2 -p allgpu
module purge

# set the environment. Alternatively use module load maxwell fbpic, see below
module load maxwell openmpi/4.0.3 cuda/10.2
export VIRTUAL_ENV=/software/fbpic/0.18
export PATH=/software/fbpic/0.18/bin:$PATH

# run as an mpi-job
mpirun -mca pml ucx -n 2 -npernode 2 python3 lwfa_script.py
exit

Running fbpic in batch

cd /beegfs/desy/user/schluenz/fbpic
sbatch fbpic.batch.sh

# The corresponding batch-script looks like this:
#!/bin/bash
#SBATCH --time=0-00:05:00
#SBATCH --partition=allgpu
#SBATCH --constraint="V100|P100"
#SBATCH --nodes=2
unset LD_PRELOAD                    # when submitting from display nodes
source /etc/profile.d/modules.sh    # make sure modules work
module purge                        # clean environment
module load maxwell fbpic

rm -rf diags/
mpirun -mca pml ucx -n 2 -npernode 2 python3 lwfa_script.py

# the output:
# the errors can be savely ignored
ERROR: ld.so: object 'libdlfaker.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object 'libvglfaker.so' from LD_PRELOAD cannot be preloaded: ignored.

# message from loading modules
CUDA 10.2 loaded

# output from the program
FBPIC (0.18.0)

Running on GPU with 2 MPI processes 

Initializing laser pulse on the mesh...
Done.

|███████████████████████████████████| 1800/1800, 0:00:00 left, 16 ms/step
Total time taken (with compilation): 0:00:54
Average time per iteration (with compilation): 30 ms

# can also be ignored
NumbaWarning: /software/fbpic/0.18/lib64/python3.6/site-packages/numba/np/ufunc/parallel.py:363:
The TBB threading layer requires TBB version 2019.5 or later i.e., TBB_INTERFACE_VERSION >= 11005. Found TBB_INTERFACE_VERSION = 6103. The TBB threading layer is disabled.