What is a Container?#

A container is a unit of software which packages up all the code, along with its dependencies, so that it can run quickly, independently, and reliably in any computing environment, including local machine, physical data centers, and public and private cloud infrastructures.

Each container has a container image, which is a standalone, lightweight, executable package that includes all the essentials to run the containerized application. This includes the code, runtime, system tools, and system libraries.

Docker, Singularity/Apptainer#

Both Docker and Singularity/Apptainer are software programs used to create containers. While Docker is more widely used it has some safety issues, which is why Singularity/Apptainer is often used to bring containers and reproducibility to scientific computing on high-performance computing (HPC) systems.

Running QuNex using a container significantly simplifies the installation and administration of the environment and enables quick deployment of QuNex. This page presents basic information about how to use a QuNex container. QuNex can be employed both using a Docker or a Singularity / Apptainer container. Note that although Singularity and Apptainer seem different solution, Apptainer is a fork of Singularity and for practical use they are mainly interchangeable. Although the specifics can differ slightly depending on the container one uses, we have made every effort to simplify and unify the use across container types.

This page provides the following information:

  • How to install a Docker container

  • How to install a Singularity/Apptainer container

  • How to run commands from within a container

  • How to run QuNex using qunex_container command

  • How to run QuNex using shell scripts

  • How to schedule QuNex container on a cluster using qunex_container

  • How to manage QuNex environment within a container

Deploying the Docker container#

To deploy and use the QuNex Docker container please follow the QuNex quick start.

Deploying the Singularity/Apptainer container#

Many computer environments (e.g. high performance university clusters) do not allow or enable the use of Docker containers due to permission issues. In this case a Singularity/Apptainer container may be used. To use a Singularity/Apptainer container you will need to have a working Singularity/Apptainer environment. Consult official documentation on how to set it up and use it (Singularity, Apptainer).

To deploy a Singularity/Apptainer container you have two options, you can download a prebuilt container from our QuNex container repository or you can build the Apptainer image yourself. For both options to work, make sure you have registered for QuNex and have access to the QuNex container image via the QuNex GitLab.

You can download the built Singularity/Apptainer container in two ways. The first option is to click on the qunex_suite-<tag>.sif file in the browser, the second option is by cloning the repository. To clone this repository and the Singularity/Apptainer image you need to configure Git Large File Storage (Git LFS), for instructions about how to do this, please consult https://git-lfs.github.com/.

The second option of deploying a Singularity/Apptainer container is to build it yourself. The instructions for this process are listed below.

# export your username for the QuNex container registry
# replace <username> with your username. You can see you username at https://gitlab.qunex.yale.edu/-/profile
# if you use SINGULARITY instead of APPTAINER, use SINGULARITY_DOCKER_USERNAME
export APPTAINER_DOCKER_USERNAME=<username>

# set the version
TAG=<stable_container_tag>

# build the image
apptainer build --docker-login qunex_suite-${TAG}.sif docker://gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:${TAG}

The Singularity/Apptainer build will take some time to run (it can take up to 5 hours), so if you are building on a HPC system you will most likely have to use a compute node instead of a login node with an interactive terminal session. How to do this depends on the HCP system at your institution. If you do not know what to do your system admins at your institution should be able to help you.

Another issue you might run into is insufficient space for Singularity/Apptainer cache (you need 50GB+ of disk space). This happens because the default location usually does not have enough allocated space by the system administrators. To overcome this, change the location of the apptainer's cache to a location on the disk where you are able to allocate sufficient space (e.g. a scratch space or your home space).

# Set apptainer cache to scratch space
export APPTAINER_CACHEDIR=/scratch/<my_username>/apptainer

# Set singularity cache to home dir
export APPTAINER_CACHEDIR=~/apptainer

Running QuNex commands through the container#

There are two options to run QuNex commands through the container. The recommended one uses the qunex_container script, a special script we prepared to ease the use of container. The alternative is to enter the container and work within it, this is recommended for more advanced users that already have experience with using containers. Both options are described below.

Running the container by using the qunex_container script#

This is the heavily advised and the easiest way to use QuNex and also the most extensively tested and robust. For extensive documentation on this please refer to this Wiki page.

Entering a container interactively#

QuNex container can also be used by running commands within the container interactively. This requires starting up the container and entering the system in it. As mentoined before using the qunex_container script is advised, however for more advanced users, this approach can also come in handy as you often have better control and visibility of what is happening.

Entering a Docker container#

Here is the terminal commands to enter into the Docker container:

docker run -it -v /path/on/local:/path/in/container gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:<qunex_version> bash

The -it flag stands for interactive mode. Please see all flags/options here: Docker container run options. Note that to be able to access and process the data, you will need to mount the folder with the data on the container.

The -v parameter to mount a folder on your machine to the container and thus give access to data to the system in the container.

Entering a Singularity/Apptainer container#

Below is an example command to enter a Singularity/Apptainer container, here the --bind parameter can be used to mount a folder on your machine to the container.

apptainer exec --bind /path/on/local:/path/in/container qunexcontainer-<qunex_version>.sif bash