Introduction

Containers provide an easy and reproducible way of running your workloads. Docker is a popular format, but it is unavailable on Maestro due to security issues. Instead, you should use Apptainer (formerly Singularity).

To check what is installed, do:

apptainer (bash)

(0)-(kpetrov@maestro-submit:~)->module av apptainer
----------------------------------------------- /opt/gensoft/devmodules ------------------------------------------------
apptainer/1.0.0  apptainer/1.1.3  apptainer/1.1.5  apptainer/1.1.9
apptainer/1.0.3  apptainer/1.1.4  apptainer/1.1.6  apptainer/1.2.2(default)

We make frequent security updates, so you should always load the default version with:

apptainer (bash)

(0)-(kpetrov@maestro-submit:~)->module load apptainer

Building your container

Before submitting jobs, you should always ensure the container image is already constructed on Maestro.  Containers are identified by their vendor and image name, so you can start from any Docker container. First, make sure the image is on your disk:

Code Block (text)

apptainer build localimage.simg  docker://vendor/image

apptainer run <suitable mounts> localimage.simg command

and never this:

Code Block (text)

apptainer run docker://vendor/image

Most likely you should bind /pasteur and /local in the container. The home directory is mounted automatically. This may lead to unexpected behaviour with local Python packages, installed in your home outside the container. To ensure that only the packages inside the container will be used, do:

Code Block (text)

export PYTHONNOUSERSITE=yes

If you need to make changes to the container,  check the example at the end of this page.

Running your container

If you are using NextFlow, please refer to its documentation here, as it contains important information.

Many containers available online claim that they need a GPU to run. This is not always the case. To let a container to use a GPU you have to use "-p gpu -q gpu --gres=gpu:1" as your srun parameters.

Moreover, do not forget the "–nv" option to let apptainer know it should look for an NVIDIA card. Once your workflow is completed, you should check that your code actually utilized the GPU, by using

Code Block (bash)

reportseff -g <JobID>

which will report how efficiently you have used the GPU. If the numbers under "GPU" are zeroes, you should run your container with "-p common,dedicated -q fast" and without "-nv"

Let us look in practice at a container provided by SVRTK

Code Block (bash)

module load apptainer 
apptainer build svrtk.apt fetalsvrtk/svrtk:general_auto_amd

It claims to use GPU and expects the data in /home/data, and we have it elsewhere, so we do the following:

Code Block (bash)

srun -p gpu -q gpu --gres=gpu:1 apptainer run --nv --writable-tmpfs --mount 'type=bind,source=/pasteur/helix/projects/hpc/kpetrov/SVRTK/data,destination=/home/data'  svrtk.apt bash /home/auto-proc-svrtk/scripts/auto-thorax-reconstruction.sh /home/data/001-0176-O-S /home/data/out-thorax-recon-results 1 3.0 0.8 1

This will fail with an error message saying it cannot write to  /home/tmp_proc, so we will also mount that directory.

Code Block (bash)

srun -p gpu -q gpu --gres=gpu:1 apptainer run --nv --writable-tmpfs --mount 'type=bind,source=/pasteur/helix/projects/hpc/kpetrov/SVRTK/data,destination=/home/data' --mount 'type=bind,source=/pasteur/helix/projects/hpc/kpetrov/SVRTK/tmp,destination=/home/tmp_proc' svrtk.apt bash /home/auto-proc-svrtk/scripts/auto-thorax-reconstruction.sh /home/data/001-0176-O-S /home/data/out-thorax-recon-results 1 3.0 0.8 1

Note that the container would be able to write to the directories you mount. Here, for example, it will remove everything in the SVRTK/tmp directory, so never mount your HOME or Project directory, make a suitable subdirectory instead. 

This would run in about half an hour. Once it is finished, wait a bit and check if it used the GPU:

Code Block (bash)

(0)-(kpetrov@maestro-submit:/pasteur/helix/projects/hpc/kpetrov/SVRTK)->reportseff -g 4643217
JobID               State       Elapsed  TimeEff   CPUEff   MemEff   GPUEff   GPUMem 
4643217           COMPLETED    00:34:12   2.4%     98.0%     83.2%     0.0%     0.0%  
  maestro-3017                                     98.0%     83.2%     0.0%     0.0%

So it didn't, and the execution time was reasonable, so we will continue running it on common/dedicated partitions:

Code Block (bash)

srun -p common,dedicated -q fast apptainer run --writable-tmpfs --mount 'type=bind,source=/pasteur/helix/projects/hpc/kpetrov/SVRTK/data,destination=/home/data' --mount 'type=bind,source=/pasteur/helix/projects/hpc/kpetrov/SVRTK/tmp,destination=/home/tmp_proc' svrtk.apt bash /home/auto-proc-svrtk/scripts/auto-thorax-reconstruction.sh /home/data/001-0176-O-S /home/data/out-thorax-recon-results 1 3.0 0.8 1

And check the efficiency

Code Block (text)

JobID        State       Elapsed  TimeEff   CPUEff   MemEff   GPUEff   GPUMem 
5145661    COMPLETED    00:33:58   2.4%     98.5%    83.1%     ---      ---

Custom container

A typical recipe file is(here we build a container for VNC):

apptainer (bash)

Bootstrap: docker
From: consol/rocky-icewm-vnc

%labels
Author: here goes your name and Pasteur email

%files
#list of files which are required to build

%post
# Use all available cores for compilation
export MAKEFLAGS="-j$(nproc)"

# Enable powertools
dnf install -y dnf-plugins-core

# Upgrade packages to most recent versions
dnf -y upgrade
dnf -y install epel-release
dnf -y --enablerepo=epel group
dnf -y group install "Xfce"
dnf -y group install "System Tools"

# Install additional packages
dnf -y install patch git wget cmake libsndfile \
ncurses-devel libxkbcommon-x11 
dnf clean all