What is a Container?
What is a Container?#
A container is a unit of software which packages up all the code, along with its dependencies, so that it can run quickly, independently, and reliably in any computing environment, including local machine, physical data centers, and public and private cloud infrastructures.
Each container has a container image, which is a standalone, lightweight, executable package that includes all the essentials to run the containerized application. This includes the code, runtime, system tools, and system libraries.
What is Docker / Apptainer (Singularity)?#
Both Docker and Apptainer are software programs used to create containers. While Docker is more widely used, Apptainer is often used to bring containers and reproducibility to scientific computing and high-performance computing (HPC).
Running QuNex using a container significantly simplifies the installation and administration of the environment and enables quick deployment of QuNex. This page presents basic information about how to use a QuNex container. QuNex can be employed both using a Docker or a Apptainer container. Although the specifics can differ slightly depending on the container one uses, we have made every effort to simplify and unify the use across the two container types.
This page provides the following information:
How to install a Docker container
How to install an Apptainer container
How to run commands from within a container
How to run QuNex using
How to run QuNex using shell scripts
How to schedule QuNex container on a cluster using
How to manage QuNex environment within a container
Deploying the Docker container#
To deploy and use the QuNex Docker container please follow the QuNex quick start.
Deploying the Apptainer container#
Many computer environments (e.g. high performance university clusters) do not allow or enable the use of Docker containers due to permission issues. In this case a Apptainer container may be used. To use a Apptainer container you will need to have a working Apptainer environment. Consult Apptainer documentation on how to set it up and use it.
To deploy a Apptainer container you have two options, you can download a built container from our QuNex container repository or you can build the Apptainer image yourself. For both options to work, make sure you have registered for QuNex and have access to the QuNex container image via the QuNex GitLab.
You can download the built Apptainer container in two ways. The first option is to click on the
qunex_suite-<tag>.sif file in the browser, the second option is by cloning the repository. To clone this repository and the Apptainer image you need to configure Git Large File Storage (Git LFS), for instructions about how to do this, please consult https://git-lfs.github.com/.
The second option of deploying a Apptainer container is to build it yourself. The instructions for this process are listed below.
# First export your username for the Docker repository for QuNex to be visible to the Linux environment. # Do not edit the name of the variable below as Apptainer expects this convention # Where you replace <username> with your username. You can get it, or change it at https://gitlab.qunex.yale.edu/-/profile/ account. export APPTAINER_DOCKER_USERNAME=<username> # Build the image # Below replace the tag with the latest container tag apptainer build --docker-login qunexcontainer-<stable_container_tag>.sif docker://gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:<stable_container_tag>
The Apptainer build will take some time to run (it can take up to 5 hours), so if you are building on a HPC system you will most likely have to use a compute node instead of a login node with an interactive terminal session. How to do this depends on the HCP system at your institution. If you do not know what to do your system admins at your institution should be able to help you.
Another issue you might run into is insufficient space for Apptainer cache (you need 50GB+ of disk space). This happens because the default location usually does not have enough allocated space by the system administrators. To overcome this, change the location of the apptainer's cache to a location on the disk where you are able to allocate sufficient space (e.g. a scratch space or your home space).
# Set apptainer cache to scratch space export APPTAINER_CACHEDIR=/scratch/<my_username>/apptainer # Set singularity cache to home dir export APPTAINER_CACHEDIR=~/apptainer
Running QuNex commands through the container#
There are two options to run QuNex commands through the container. The default one uses the
qunex_container script, we prepared this script to ease the use of container. We recommend that you use the QuNex container through this script. The alternative is to enter the container and work within it. Both options are described below.
Running the container by using the qunex_container script#
For extensive documentation of this please refer to this Wiki page.
Entering the Docker container#
QuNex container can also be used by running commands within the container interactively. This requires starting up the container and 'logging in'. Both Docker and Apptainer use the
run command to enter into a container. Here is the terminal commands to enter into the Docker container:
docker container run -it gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:<stable_container_tag> bash
-it flag stands for interactive mode. Please see all flags/options here: Docker container run options. Note that to be able to access and process the data, you will need to mount the folder with the data on the container.
Use the -v flag to mount a folder on your machine to the container.
docker container run -it -v /path/on/local:/path/in/container gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:<stable_container_tag> bash
Entering the Apptainer container#
apptainer run qunexcontainer-<stable_container_tag>.sif