Singularity to Apptainer renaming
Singularity has been renamed to Apptainer in 2022 due to legal constraints. In general, just the name has changed and all options are the same. Replacing the command `singularity` with `apptainer` should work on updated systems.
Pulling remote Container Images to local Directories
While Apptainer/Singularity can pull directly from a remote container repository like Dockerhub, it is advisable to pull an container image into a local directory and run Apptainer/Singularity directly on it.
Background is, that Apptainer/Singularity pulls on demand a remote image into the local cache directory, which is normally under $HOME/.singularity - since this is on AFS parallel access can be quite slow. So explicitly pulling a container onto DUST as a faster, scalable file system will be more performant for batch jobs etc. (plus your $HOME dir don't get filled up with container files - see Apptainer/Singularity Cache and TMP Directories how to put the tmp/cache dirs somewhere else)
Putting a container on a DUST path
To build a local container image on DUST from a remote hub, run something like
> singularity build --sandbox /nfs/dust/your/path/on/dust/container.d/name.d docker://gitlab-registry.cern.ch/group/some-container:latest
with the '–sandbox' flag this will create a container image on DUST, i.e., an expanded directory tree instead of a single file blob (might be advantageous if you want to play around in the directory from the outside or for performance reasons as the DUST filesystem can balance individual files instead of a large file blob)
If you notice problems with mounting the container image directory (might happen for some network file system constellations), you can try to re-build the container image without the --sandbox flag. In that case you will get a compressed single image file.
If the image build fails due to missing free space, it might be that for large cotnainers there is not enough space in your Apptainer/Singularity temp directory - that is by default in your $HOME directory. In that case, follow the Apptainer/Singularity: Image Cache documentation on how to move Apptainer/Singularity's temporary and cache directories.
To run the local container just point Apptainer/Singularity to the directory, e.g.,
> singularity shell/exec/run/... {–-contain} /nfs/dust/your/path/on/dust/container.d/name.d
CERN gitlab
Credentials for connecting to a Dockerhub or a Apptainer/Singularityhub requiring authentication
If a container hub requires an authentication, you can either hard-code them in your container recipe (see below) or export them as environment variables, so that Apptainer/Singularity knows whom it represents
export SINGULARITY_DOCKER_USERNAME=yourusername
export SINGULARITY_DOCKER_PASSWORD=yourpassword
Bootstrapping a new container
To bootstrap a container from CERN's gitlab instance as base for building an own container, add to your Apptainer/Singularity recipe the gitlab endpoint with the right project (here: maxcw) and ContainerName:Version
> cat Singularity
Bootstrap: docker
From: docker://gitlab-registry.cern.ch/maxcw/dockerhandons:latest
Username: YOURCERNUERNAME
Password: YOURCERNPASSWORD...your Singularity recipe additions here
which you can build into your own container
> sudo singularity build --sandbox MyGitlabTest.d Singularity
Building into existing container: test
...
Docker image path: gitlab-registry.cern.ch/maxcw/dockerhandons:latest
Cache folder set to /root/.singularity/docker
[5/5] |===================================| 100.0%
Exploding layer: sha256:7ab860fcccec4790c06571131eed7a3a29f3d46e271a20ed4861025f3ab9c58b.tar.gz
...
Cleaning up...
> singularity shell MyGitlabTest.d
Singularity MyGitlabTest :~> ...
Running a remote container
To jump directly into a container from a remote repository, you can give the address as the name, e.g.,
> singularity shell --bind /cvmfs:/cvmfs --scratch /var/tmp,/tmp docker://gitlab-registry.cern.ch/maxcw/dockerhandons:latest
(the images you are pulling this way end up cached locally under ${HOME}/.singularity - if you pull a lot containers, watch out for the sizes of locally cached files)
If you are using a container regularly, it might be better to bootstrap the container image first to have it accessible somewhere in your local environment.