Computing : Problems in Containers with 'No space left on device' due to limited TMP directories

Singularity to Apptainer renaming

Singularity has been renamed to Apptainer in 2022 due to legal constraints. In general, just the name has changed and all options are the same. Replacing the command `singularity` with `apptainer` should work on updated systems.

Problem:

For example, you want to use a Apptainer/Singularity container in the shell mode (with the contain flag set) to compile (within the container) a program. However, your compilation fails with an error like

error: error writing to /tmp/cceNuA3r.s: No space left on device

but (outside) the container you have sufficient space in your $HOME directory etc.

(for problems with not enough space while building a container see Apptainer/Singularity Cache and TMP Directories )

Solution

Your compilation needs sufficient space in temporary directories like /tmp (in the container) or /var/tmp (in the container) for some intermediary files.
However, Apptainer/Singularity limits by default the available space in /tmp or /var/tmp to avoid problems, where container runs clutter temporary files.

So, we will start the Apptainer/Singularity container and point its temporary directories to directories beneath, that have more space available (there are also other options available, but we go for the bind-mount for educational reasons).

create explicitly TMP directories under your DUST scratch path (or some other file system, where you have sufficient free space

 mkdir -p /nfs/dust/belle2/user/YOURUSERNAME/scratch/tmp
 mkdir -p /nfs/dust/belle2/user/YOURUSERNAME/scratch/var/tmp

run the Apptainer/Singularity container and bind-mount these directories onto /tmp and /var/tmp in the container with the flag --bind SOURCE:DESTINATION

singularity shell --bind /nfs/dust/belle2/user/YOURUSERNAME/scratch/tmp:/tmp --bind /nfs/dust/belle2/user/YOURUSERNAME/scratch/var/tmp:/var/tmp   ...YOUR_OTHER_SINGULARITY_OPTIONS..

Now all file operations under /tmp or /var/tmp (in the container) would end up (outside the container) on your DUST directories

Warning

Do not share these directories between containers. If container X and container Y would use the same temporary directories, nasty things can be expected to happen!

Alternatively

With the '--workdir /path/to/DUST/dir' flag, the directories (in the container) /tmp, /var/tmp and $HOME are put into this workdir directory (on the outside) by Apptainer/Singularity already.