Maxwell : micromamba EL9 container

Some python module won't work on Centos_7 anymore. In such cases you can create a conda environment for RedHat EL9 using singularity images, and run them either under Centos_7 or EL9.

Building a container from scratch

The best option is to create your own image, copy it to /beegfs or upload it to the container registry.  A sample is available under /beegfs/desy/sys/singularity/micromamba-el9/.  If you have a linux desktop you can create the singularity image directly on your desktop:

# some preparations on maxwell, only need to be done once
ssh max-wgs # for example
mk-beegfs # if you don't have a beegfs-folder yet
mkdir -p /beegfs/desy/user/$USER/singularity /beegfs/desy/user/$USER/stmp /beegfs/desy/user/$USER/scache
exit

# on your desktop
scp -r max-wgs:/beegfs/desy/sys/singularity/micromamba-el9/ .
cd micromamba-el9
# modify, rename micromamba-el9.def to include the python modules you need for your project.
# virtualGL is included in the image to be able running graphical applications on max-display using FastX
singularity build --fakeroot micromamba-el9.sif micromamba-el9.def

scp micromamba-el9.sif max-wgs:/beegfs/desy/user/$USER/singularity/

# back on maxwell
ssh max-display # for example 

# particularly on share nodes like max-wgs, max-display /tmp or /scratch are quickly becoming too small. wont't hurt adding this to ~/.bashrc or ~/.zshrc
export SINGULARITY_TMPDIR=/beegfs/desy/user/$USER/stmp
export SINGULARITY_CACHEDIR=/beegfs/desy/user/$USER/scache 

# the home-directory is automatically imported into the singularity container. You might want to have other places included, for example
export SARGS="-B /asap3:/asap3,/beegfs/desy/$USER:/beegfs/desy/$USER"

# you don't want to include anything from ~/.local
export PYTHONNOUSERSITE=1

# enter the image and run python
singularity exec $SARGS /beegfs/desy/user/$USER/singularity/micromamba-el9.sif /opt/micromamba/bin/python 
Python 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:39) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

# the OS inside the container:
singularity exec /beegfs/desy/user/$USER/singularity/micromamba-el9.sif bash
Apptainer> cat /etc/os-release 
NAME="Rocky Linux"
VERSION="9.3 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.3 (Blue Onyx)"

So you have your entire python environment inside the image. The filesystem only operates on a single file, and you haven't modified anything in your home-directory which presumably contains quite a bit of Centos_7 specific apps and code.

Building a singularity image directly on Maxwell will work exactly the same way, but requires some administrative preparation (subuid). Please let us know in case you'd like to build images on Maxwell.

Building singularity images directly on Maxwell

the above instructions with minimal changes:

# some preparations on maxwell, only need to be done once
ssh max-wgs # for example
mk-beegfs # if you don't have a beegfs-folder yet
mkdir -p /beegfs/desy/user/$USER/singularity /beegfs/desy/user/$USER/stmp /beegfs/desy/user/$USER/scache

cd /beegfs/desy/user/$USER/singularity 
cp -r max-wgs:/beegfs/desy/sys/singularity/micromamba-el9/ .
cd micromamba-el9

# modify, rename micromamba-el9.def to include the python modules you need for your project.
# virtualGL is included in the image to be able running graphical applications on max-display using FastX
singularity build --fakeroot micromamba-el9.sif micromamba-el9.def

cp micromamba-el9.sif max-wgs:/beegfs/desy/user/$USER/singularity/

# the home-directory is automatically imported into the singularity container. You might want to have other places included, for example
export SARGS="-B /asap3:/asap3,/beegfs/desy/$USER:/beegfs/desy/$USER"

# you don't want to include anything from ~/.local
export PYTHONNOUSERSITE=1

# enter the image and run python
singularity exec $SARGS /beegfs/desy/user/$USER/singularity/micromamba-el9.sif /opt/micromamba/bin/python 
Python 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:39) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

# the OS inside the container:
singularity exec /beegfs/desy/user/$USER/singularity/micromamba-el9.sif bash
Apptainer> cat /etc/os-release 
NAME="Rocky Linux"
VERSION="9.3 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.3 (Blue Onyx)"

Using a pre-built image and create isolated conda-environments in your home directory

# Run the EL9 image, but redefine HOME. 
cp /beegfs/desy/sys/singularity/micromamba-el9/setup.sh ~/EL9/

# run the micromamba image with bash
singularity exec --nv --home $HOME/EL9 /beegfs/desy/sys/singularity/micromamba-el9/micromamba-el9.sif bash
source ~/setup.sh
micromamba create --prefix=~/envs/test -c conda-forge python=3.10
micromamba activate ~/envs/test   
micromamba install -c conda-forge numpy # etc

exit

Using the conda-environment on EL7 nodes

On EL7 nodes you will usually need to use the singularity image, though python itself will work. Make sure you don't have a mamba/conda setup predefined,

export PYTHONNOUSERSITE=1 # don't pull packages from ~/.local
singularity exec --nv /beegfs/desy/sys/singularity/micromamba-el9/micromamba-el9.sif $HOME/EL9/envs/test/bin/python

Using the conda-environment on EL9 node

On EL9 nodes you can safely run python directly or through singularity:

export PYTHONNOUSERSITE=1 # don't pull packages from ~/.local

# run directly 
$HOME/EL9/envs/test/bin/python

# run through singularity
singularity exec --nv /beegfs/desy/sys/singularity/micromamba-el9/micromamba-el9.sif $HOME/EL9/envs/test/bin/python