General overview#

qunex_container is an independent, self contained python3 compatible command that enables running commands against either Docker or Apptainer (Singularity) QuNex container. The command is available as part of the QuNex suite. To make use of the qunex_container, you will have to save a copy of the command to a folder that is in your PATH (e.g., /usr/local/bin or /usr/bin) or add the location of the qunex_container to your PATH. You can refer to the QuNex quick start for details about how to download and install this script.

We suggest adding the following line to your .bash_profile or similar shell initialization script:

export PATH=<path to folder with qunex_container>:$PATH

To be sure that you have the command available, you can inspect the PATH by running echo $PATH or the result of which qunex_container commands.

Once you have qunex_container installed, you can run QuNex commands the same way you would run them inside the container or if QuNex was installed locally, with two differences. First, replace:

qunex <command> [<parameters>]

with

qunex_container <command> [<parameters>]

When running qunex_container you will need to provide information about the location of the container you want to run the command against. To do this you just set the container parameter. An example use in this case would be:

qunex_container hcp_freesurfer \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1,sub2,sub3" \
    --sessionsfolder="/myStudy/sessions" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --overwrite="yes"

In the example above some of the parameters are related to the qunex_container command (e.g., container) while others are related to the hcp_freesurfer command (e.g., sessionids, overwrite, etc.). QuNex automatically distinguishes these parameters and properly propagates them to relevant commands.

The second possibility is to provide the information about the location of the container by setting an environmental variable QUNEXCONIMAGE, possibly in your .bash_profile or another environment script, e.g.:

  • For the Docker container:

export QUNEXCONIMAGE="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:<stable_container_tag>"

  • For the Apptainer container:

export QUNEXCONIMAGE="/<path_to_container_image>/<container image name>.sif"

If you want to run a set of commands within a container, instead of a single command, you can make use of the run_turnkey QuNex commands. Please consult QuNex quick start and QuNex 'turnkey' workflow for a detailed description of this.

Another option is to write a script with commands you wish to run within the container and pass the script to the container using the script parameter, e.g:

qunex_container \
    --script="/myStudy/processing/myScript.sh" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif"

For more detailed information about the use of the qunex_container command and how to pass additional settings for the container, consult the inline help by running qunex_container.

Binding/mapping external folders or setting additional container parameters#

When using a singularity container you can bind external folders by using the bind parameter, this is analog to the singularity's -B flag. To bind multiple folders use a comma separated list:

qunex_container hcp_freesurfer \
    --batchfile="/my_study/processing/batch.txt" \
    --sessions="sub1,sub2,sub3" \
    --sessionsfolder="/my_study/sessions" \
    --bind="/my_data/data:/data,/my_data/scripts:/scripts"
    --container="/my_singularity_containers/qunexcontainer-latest.sif" \
    --overwrite="yes"

When using a docker container, QuNex automatically binds the folder you are currently in as /data inside the container. To change this mapping you can use the dockeropt parameter. The dockeropt is string that lists the additional options to be used when running the Docker container. If the parameter is ommited, the content of QUNEXDOCKEROPT environment variable will be used. If neither is specified, the container will mount the current as data by specifying the following options -v "$(pwd)":/data. The parameters are to be specified in a string exactly as they would be on a command line, e.g. to run in detached mode and mount a specific folder use

qunex_container hcp_freesurfer \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1,sub2,sub3" \
    --sessionsfolder="/myStudy/sessions" \
    --dockeropt="-d -v /data/host_directory/qx_study/:/my_study/" \
    --container="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:<tag>" \
    --overwrite="yes"

Using CUDA libraries with an Apptainer container#

When using CUDA (GPU processing) inside an Apptainer (Singularity) container, the --nv flag is often required so CUDA drivers and libraries are properly setup. To achieve this, use the --nv flag in the qunex_container call:

qunex_container hcp_diffusion \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1" \
    --sessionsfolder="/myStudy/sessions" \
    --nv \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --overwrite="yes"

Also when scheduling you have to use partitions that have GPU nodes, these partitions usually have a _gpu suffix in their name.

If you are using Apptainer (Singularity) and a CUDA version that is not the QuNex's default, you also need to provide QuNex with the path to the location of the local CUDA libraries:

qunex_container hcp_diffusion \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1" \
    --sessionsfolder="/myStudy/sessions" \
    --nv \
    --cuda_path="/usr/local/cuda-11/" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --overwrite="yes"

Checking QuNex environment status inside the container#

You can use the --env_status flag to check whether everything is OK with the QuNex environment inside the container. This comes in handy when troubleshooting execution of various commands, the first thing one should check is if the QuNex environment is properly set up.

qunex_container \
    --env_status \
    --container="/myApptainerContainers/qunexcontainer-latest.sif"

Executing custom bash commands#

In the example above we executed the hcp_diffusion command within the container by using the following command:

qunex_container hcp_diffusion \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1" \
    --sessionsfolder="/myStudy/sessions" \
    --nv \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --overwrite="yes"

Sometimes you would like to execute some additional bash commands, either before entering the container or when already inside the container but before executing the actual QuNex command (hcp_diffusion in the example above). You can do that by using the bash_pre and bash_post parameters. Commands specified in the bash_pre parameter will execute before entering the container while those specified in the bash_post parameter will execute when entering the container, but before executing the actual QuNex command. In the example below we first move to a certain directory and load the singularity module (this is done via the bash_pre parameter), next we enter the QuNex container, then we set the value of an environmental variable (FSL_FIXDIR) via the bash_post parameter, finally we execute the hcp_diffusion command.

qunex_container hcp_diffusion \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1" \
    --sessionsfolder="/myStudy/sessions" \
    --nv \
    --bash_pre="cd /studies/qunex/;module load singularity" \
    --bash_post="export FSL_FIXDIR=/studies/qunex/fix" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --overwrite="yes"

As you can see multiple bash commands can be chained together by using the semicolon character.

Executing custom bash scripts in the QuNex environment#

Inside the QuNex container you will find many neuroimaging tools that are fully installed and preconfigured. A common scenario is that one might need to use this tools directly for some processing or analysis since QuNex does not cover all the possible use cases for these tools. For this purpose you can use the qunex_container script to execute a custom bash script within the container. To achieve this you simply need to prepare a bash script that includes the desired commands and make that script executable. Next, we pass the script to qunex_container with the script parameter.

For example, if I want to merge two images using fslmerge I would prepare this script:

#!/bin/bash

flsmerge -t /data/mri_data/merged.nii.gz /data/mri_data/image1.nii.gz /data/mri_data/image2.nii.gz

Let's say the script is saved as /data/mri_scripts/flsmerge.sh. To execute it using FSL in the QuNex container I would then run:

qunex_container \
    --script="/data/mri_scripts/flsmerge.sh" \
    --container="/data/qx_containers/qunex_suite-0.98.0.sif"

If I were to add the scheduler parameter execution of my custom script will be scheduled on a HPC cluster.

Scheduling commands to run against the QuNex container on a cluster#

To make use of the parallelization in a cluster environment, qunex_container can schedule running the specified QuNex command as individual jobs to multiple computer cluster nodes, so that each job will process only one, or a set of sessions (or subjects) in parallel with all the others. Currently PBS and SLURM scheduling systems are supported.

Scheduling is enabled by specification of additional parameters to the qunex_container command. First, to use a scheduler, provide a scheduler settings string to the scheduler parameter. The general information about the settings string is available as inline documentation by running qunex_container, whereas more detailed information is available by running qunex_container ?scheduler.

An example of scheduling qunex_container to a single node:

qunex_container hcp_freesurfer \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1,sub2,sub3" \
    --sessionsfolder="/myStudy/sessions" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --scheduler="SLURM,time=2-00:00:00,cpus-per-task=3,mem-per-cpu=20GB,partition=week" \
    --overwrite="yes"

This command will schedule the QuNex hcp_freesurfer command to be run as a single job with three sessions to process.

To distribute commands over multiple jobs and nodes to run in parallel, you have to provide two parameters: 1. sessions, and 2. parsessions. The first parameter provides a list of sessions (subjects) that need to be processed, the second parameter provides information on how many sessions are to be run in a single job. qunex_container will then schedule as many jobs as necessary to process all the sessions, so that at most N (N being equal to parsessions) are scheduled per job. An example of scheduling multiple jobs is:

qunex_container hcp_freesurfer \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessionsfolder="/myStudy/sessions" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --parsessions="4" \
    --scheduler="SLURM,time=2-00:00:00,cpus-per-task=4,mem-per-cpu=16GB,partition=week" \
    --overwrite="yes"

If there are 202 sessions listed in the /myStudy/processing/batch.txt file, then the upper command will schedule 51 jobs, each job running the hcp_freesurfer command against the QuNex container. The first 50 jobs will run 4 sessions in parallel within a container. The last job will run the remaining 2 sessions in parallel within a container.

For more specific information, please see the qunex_container inline help by running qunex_container.

NOTE: Passing --sessions parameter values cleanly

In the case that --sessions parameter needs to be passed to a command, but not used to split up processing into multiple container calls (e.g., in import_dicom or import_bids commands), specify the sessions to pass to the call using the --csessions parameter. The value of that parameter will be passed on as --sessions parameter to the command.

Scheduling example#

qunex_container import_dicom \
    --sessionsfolder="/Studies/MyStudy/sessions/" \
    --csessions="ap9374" \
    --masterinbox="none" \
    --check="any" \
    --overwrite="no" \
    --container="/software/Apptainer/qunexcontainer-latest.sif" \
    --scheduler="SLURM,time=0-00:20:00,cpus-per-task=1,mem-per-cpu=16GB,partition=day"

In the above case, only one container will be scheduled and within the container the import_dicom command will be run with --sessions="ap9374".

Making changes to the QuNex container environment#

This section is for more advanced users that will sometime want to make some changes to the OS and QuNex environments inside the container. For example, through described methods you can force QuNex inside container to use libraries that are located outside of the container, on your system. Doing so, you can use versions of libraries and tools (e.g. HCP Pipelines, FSL, etc.) that are different from the ones officially packaged into the container. One option is to use the previously mentioned bash_post parameter to execute bash code within the container. Some additional options that provide further flexibility are outlined below.

QuNex environment#

Once inside the container you can run qunex commands. Try running qunex to see the help output, or qunex_envstatus to see the environment variables. For general QuNex usage, see General Usage Overview.

Of note, the environment script is run automatically when commands are run against the container using qunex_container. When run, the script (/opt/qunex/env/qunex_environment.sh) will reset the Apptainer shell environment and clear any prior environment variables such that it reflects the default environment state for QuNex within the container. This reset is especially important for Apptainer containers as they are "permeable" to inheriting user environment variables that were set prior to entering the Apptainer shell.

There might be situations where one would like to make changes to the Apptainer shell container environment. One possibility would be to change a specific setting regarding the behavior within the container. For instance, a user can change the default options set for running octave commands, which are specified in the QUNEXMCOMMAND environment parameter. Another possibility is to use software and/or dependencies that are not explicitly part of the QuNex Container. For instance, a user can set an environment variable to point to a FreeSurfer installation outside of the QuNex Container default by changing the FREESURFERDIR environmental variable once inside the Apptainer shell. Put differently, a user can dynamically change and redeclare environment variables once inside the Apptainer shell to a use specified path.

Specifically, there are three mechanisms available to change the environment variables used within the QuNex container, presented in the following three sections.

Option I: .qunex_container.rc file#

If .qunex_container.rc file is present in the home directory, it will be sourced after the QuNex environment was reset. Notice that in the case of the Apptainer container, the home folder is the user's home folder on the host computer. In case of the Docker container the home folder is /root folder within the container. To have .qunex_container.rc present in the Docker home folder, the folder that contains the file on the host computer has to be mounted as the Docker container's /root folder.

Option II: QUNEXCONTAINERENV variable#

If QUNEXCONTAINERENV is set, then it is considered to hold the path to the environment script that should be sourced within the container. The script will be sourced after the environment is reset and after checking for the presence of the .qunex_container.rc file. Again, there are differences between the Apptainer and the Docker container use. For Apptainer, the QUNEXCONTAINERENV is set in the user environment on the host computer and the path specified is a path valid on the host computer. For the Docker container, the variable has to be passed using additional Docker options, the script has to be present in a filesystem mounted to the Docker container, and the path has to be correct from within the Docker container.

Option III: Use of con_ prefix#

After the environment was reset and the presence of .qunex_container.rc and QUNEXCONTAINERENV was tested, the environment script then checks for presence of valid QuNex environment variables prefixed with con_, e.g. con_FREESURFERDIR or con_FSLDIR. If these variables are present, the environment variables they refer to (FREESURFERDIR, FSLDIR in the previous example) will be set to their value. The formula <ENVIRONMENT VARIABLE NAME>=${con_<ENVIRONMENT VARIABLE NAME>} might in a specific environment result in:

export FREESURFERDIR=${con_FREESURFERDIR}
export FSLDIR=${con_FSLDIR}

Again, note that in the case of the Apptainer container, the variables need to be set in the host computer user environment and you should not use the --cleanenv Apptainer flag which prevents host variables from propagating into the container. When running through qunex_container you should use --nocleanenv flag to achieve this. In the case of the Docker container, the variables need to be passed when the container is run.

Order of operation for setting the environment#

The environment will be adjusted in the following order:

  1. .qunex_container.rc

  2. QUNEXCONTAINERENV

  3. con_<ENVIRONMENT VARIABLE NAME>

This means that if a specific variable, e.g. FSLDIR is set in .qunex_container.rc, and then again in the script referenced by QUNEXCONTAINERENV, the latter will overwrite the former. If it is again specified as con_FSLDIR this value will overwrite any previous value.

Setting QuNex environment variables when using qunex_container#

Container environment variables can also be specified using the --envars parameter passed to qunex_container command. Specifically, envars parameter should be a pipe separated string of "<variable>=><value>|<variable>=><value>" pairs, illustrated below:

Example call#

qunex_container hcp_freesurfer \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1,sub2,sub3" \
    --sessionsfolder="/myStudy/sessions" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --envars="FSLDIR=>/usr/local/bin/fsl|FREESURFERDIR=>/opt/bin/freesurfer-6-modified"

Note that internally, qunex_container makes use of con_* mapping so the values specified in the qunex_container --envars parameter will have precedence over all other methods.