General overview#

qunex_container is an independent, self contained python3 compatible command that enables running commands against either Docker or Singularity/Apptainer QuNex container. The command is available as part of the QuNex suite. To make use of the qunex_container, you will have to save a copy of the command to a folder that is in your PATH (e.g., /usr/local/bin or /usr/bin) or add the location of the qunex_container to your PATH. You can refer to the QuNex quick start for details about how to download and install this script.

We suggest adding the following line to your .bashrc or a similar shell initialization script:

export PATH=<path to folder with qunex_container>:$PATH

To be sure that you have the command available, you can inspect the PATH by running echo $PATH or the result of which qunex_container commands.

Once you have qunex_container installed, you can run QuNex commands the same way you would run them inside the container or if QuNex was installed locally, with two differences. First, replace:

qunex <command> [<parameters>]

with

qunex_container <command> [<parameters>]

When running qunex_container you will need to provide information about the location of the container you want to run the command against. To do this you just set the --container parameter. An example use in this case would be:

qunex_container hcp_freesurfer \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1,sub2,sub3" \
    --sessionsfolder="/myStudy/sessions" \
    --overwrite="yes" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif"

In the example above some of the parameters are related to the qunex_container command (e.g., --container) while others are related to the hcp_freesurfer command (e.g., --sessions, --overwrite, etc.). QuNex automatically distinguishes these parameters and properly propagates them to relevant commands.

Binding/mapping external folders or setting additional container parameters#

You can bind external folders by using the --bind parameter of the qunex_container script. To bind multiple folders use a comma separated list:

qunex_container hcp_freesurfer \
    --batchfile="/my_study/processing/batch.txt" \
    --sessions="sub1,sub2,sub3" \
    --sessionsfolder="/my_study/sessions" \
    --bind="/my_data/data:/data,/my_data/scripts:/scripts"
    --container="/my_singularity_containers/qunexcontainer-latest.sif" \
    --overwrite="yes"

Using CUDA in the container#

When using CUDA (GPU processing) you will have to add the --cuda flag to the qunex_container script run. This flag will sort out everything required for passing CUDA drivers into the container. For this to work, you will need to have NVIDIA Container Toolkit installed on the host system.

qunex_container hcp_diffusion \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1" \
    --sessionsfolder="/myStudy/sessions" \
    --cuda \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --overwrite="yes"

An alternative that might work with Singularity/Apptainer containers and does not require NVIDIA Container Toolkit on the host system is to use the --nv flag:

qunex_container hcp_diffusion \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1" \
    --sessionsfolder="/myStudy/sessions" \
    --nv \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --overwrite="yes"

Note that, per our tests this solution unfortunately does not work very robustly on many systems.

When scheduling you have to use partitions that have GPU nodes, these partitions usually have a _gpu suffix in their name.

Sometimes, when you are using a CUDA version that is not the QuNex's default, you might also need to provide QuNex with the path to the location of the local CUDA libraries through the --cuda_path parameter:

qunex_container hcp_diffusion \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1" \
    --sessionsfolder="/myStudy/sessions" \
    --nv \
    --cuda_path="/usr/local/cuda-11/" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --overwrite="yes"

Passing additional container options#

To pass any additional options to the container run, you can use the --containeropt parameter. The --containeropt parameter is string that lists the additional options to be used when running the container. The parameters are to be specified in a string exactly as they would be on a command line, e.g. to run in detached mode and mount a specific folder use:

qunex_container hcp_freesurfer \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1,sub2,sub3" \
    --sessionsfolder="/myStudy/sessions" \
    --containeropt="-d -v /data/host_directory/qx_study/:/my_study/" \
    --container="qunex/qunex_suite:<tag>" \
    --overwrite="yes"

Checking QuNex environment status inside the container#

You can use the --env_status flag to check whether everything is OK with the QuNex environment inside the container. This comes in handy when troubleshooting execution of various commands, the first thing one should check is if the QuNex environment is properly set up.

qunex_container \
    --env_status \
    --container="/myApptainerContainers/qunexcontainer-latest.sif"

Executing custom bash commands#

In the example above we executed the hcp_diffusion command within the container by using the following command:

qunex_container hcp_diffusion \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1" \
    --sessionsfolder="/myStudy/sessions" \
    --nv \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --overwrite="yes"

Sometimes you would like to execute some additional bash commands, either before entering the container or when already inside the container but before executing the actual QuNex command (hcp_diffusion in the example above). You can do that by using the --bash_pre and --bash_post parameters. Commands specified in the --bash_pre parameter will execute before entering the container while those specified in the --bash_post parameter will execute when entering the container, but before executing the actual QuNex command. In the example below we first move to a certain directory and load the singularity module (this is done via the --bash_pre parameter), next we enter the QuNex container, then we set the value of an environmental variable (FSL_FIXDIR) via the --bash_post parameter, finally we execute the hcp_diffusion command.

qunex_container hcp_diffusion \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="sub1" \
    --sessionsfolder="/myStudy/sessions" \
    --nv \
    --bash_pre="cd /studies/qunex/;module load singularity" \
    --bash_post="export FSL_FIXDIR=/studies/qunex/fix" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --overwrite="yes"

As you can see multiple bash commands can be chained together by using the semicolon character.

Executing custom bash scripts in the QuNex environment#

Inside the QuNex container you will find many neuroimaging tools that are fully installed and preconfigured. A common scenario is that one might need to use this tools directly for some processing or analysis since QuNex does not cover all the possible use cases for these tools. For this purpose you can use the qunex_container script to execute a custom bash script within the container. To achieve this you simply need to prepare a bash script that includes the desired commands and make that script executable. Next, we pass the script to qunex_container with the --script parameter.

For example, if I want to merge two images using fslmerge I would prepare this script:

#!/bin/bash

flsmerge -t /data/mri_data/merged.nii.gz /data/mri_data/image1.nii.gz /data/mri_data/image2.nii.gz

Let's say the script is saved as /data/mri_scripts/flsmerge.sh. To execute it using FSL in the QuNex container I would then run:

qunex_container \
    --script="/data/mri_scripts/flsmerge.sh" \
    --container="/data/qx_containers/qunex_suite-0.98.0.sif"

Scheduling commands to run against the QuNex container on a cluster#

To make use of the parallelization in a cluster environment, qunex_container can schedule running the specified QuNex command as individual jobs to multiple computer cluster nodes, so that each job will process only one, or a set of sessions (or subjects) in parallel with all the others. Currently PBS and SLURM scheduling systems are supported.

Scheduling is enabled by specification of additional parameters to the qunex_container command. First, to use a scheduler, provide a scheduler settings string to the --scheduler parameter. An example of scheduling hcp_freesurfer:

qunex_container hcp_freesurfer \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="s001,s002,s003,s004" \
    --sessionsfolder="/myStudy/sessions" \
    --overwrite="yes" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --scheduler="SLURM,time=2-00:00:00,cpus-per-task=2,mem-per-cpu=32GB,partition=week"

This command will schedule the QuNex hcp_freesurfer command. By default, QuNex will spawn a job for each session. In the above example, QuNex will spawn 3 jobs that will be executed in parallel. If you do not want such behavior, you can use the parjobs parameter to define how many jobs (at most) are spawned. For example, if I set --parjobs=1 then QuNex will spawn a single job with all 4 sessions running in it. By default the sessions in the job will be processed sequentially. You can use the --parsessions parameter to control that behavior:

qunex_container hcp_freesurfer \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="s001,s002,s003,s004" \
    --sessionsfolder="/myStudy/sessions" \
    --parjobs="2" \
    --parsessions="2" \
    --overwrite="yes" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --scheduler="SLURM,time=2-00:00:00,cpus-per-task=2,mem-per-cpu=32GB,partition=week"

Above command will spawn 2 jobs, inside each job, 2 sessions will be processed in parallel, s001 and s002 in the first job, s003 and s004 in the second job. You can also omit the --sessions parameter and all sessions in the --batchfile will be processed. For example:

qunex_container hcp_freesurfer \
    --batchfile="/myStudy/processing/batch.txt" \
    --sessions="s001,s002,s003,s004" \
    --sessionsfolder="/myStudy/sessions" \
    --parjobs="10" \
    --parsessions="5" \
    --overwrite="yes" \
    --container="/myApptainerContainers/qunexcontainer-latest.sif" \
    --scheduler="SLURM,time=2-00:00:00,cpus-per-task=2,mem-per-cpu=32GB,partition=week"

if there are 100 sessions listed in the /myStudy/processing/batch.txt file, then the upper command will schedule 10 jobs, each job running the hcp_freesurfer command against the QuNex container. Within each jobs, 10 sessions will be processed with 5 being run in parallel.

Making changes to the QuNex container environment#

This section is for more advanced users that will sometime want to make some changes to the OS and QuNex environments inside the container. For example, through described methods you can force QuNex inside container to use libraries that are located outside of the container, on your system. Doing so, you can use versions of libraries and tools (e.g. HCP Pipelines, FSL, etc.) that are different from the ones officially packaged into the container. One option is to use the previously mentioned bash_post parameter to execute bash code within the container. Some additional options that provide further flexibility are outlined below.

QuNex environment#

Of note, the QuNex environment is initialized automatically when commands are run against the container using qunex_container. When run, the script (/opt/qunex/env/qunex_environment.sh) will reset the shell environment and clear any prior environment variables such that it reflects the default environment state for QuNex within the container. This reset is especially important for Singularity/Apptainer containers as they are "permeable" to inheriting user environment variables that were set prior to entering the Singularity/Apptainer shell. By default, QuNex will run Singularity/Apptainer with the --cleanenv flag which wipes user's settings, if you want to propagate some settings, add the --nocleanenv to the qunex_container call.

There might be situations where one would like to make changes to the container environment. One possibility would be to change a specific setting regarding the behavior within the container. One possibility is to use software and/or dependencies that are not explicitly part of the QuNex Container. For instance, a user can set an environment variable to point to a FreeSurfer installation outside of the QuNex Container default by changing the FREESURFERDIR environmental variable once inside the container. Put differently, a user can dynamically change and redeclare environment variables once inside the container shell to a use specified path.

To change or set the environment variables used within the QuNex container, you can use the --bash_post parameter. By using the --bash_post parameter, we can execute custom bash commands once inside the container. For example, setting --bash_post="export FREESURFERDIR=/local/my_freesurfer" will overwrite the value of the FREESURFERDIR variable within the QuNex environment with a custom one. Sometimes the code in --bash_post can become very long, in these cases, you can prepare a custom script and then simply source it within the QuNex container. For example, I can have a script /data/qunex/qunex_container_environment.sh and in it:

export FREESURFERDIR=/local/my_freesurfer
export FSLDIR=/local/my_fsl

By setting --bash_post="source /data/qunex/qunex_container_environment.sh" above code will be executed in the container.

QuNex documentation

General overview

Contents