# General overview `qunex_container` is an independent, self contained python3 compatible command that enables running commands against either Docker or Apptainer (Singularity) QuNex container. The command is available as part of the QuNex suite. To make use of the `qunex_container`, you will have to save a copy of the command to a folder that is in your `PATH` (e.g., `/usr/local/bin` or `/usr/bin`) or add the location of the `qunex_container` to your PATH. You can refer to the [QuNex quick start](../Overview/QuickStart) for details about how to download and install this script. We suggest adding the following line to your `.bash_profile` or similar shell initialization script: ``` bash export PATH=:$PATH ``` To be sure that you have the command available, you can inspect the PATH by running `echo $PATH` or the result of `which qunex_container` commands. Once you have `qunex_container` installed, you can run QuNex commands the same way you would run them inside the container or if QuNex was installed locally, with two differences. First, replace: ``` bash qunex [] ``` with ``` bash qunex_container [] ``` When running `qunex_container` you will need to provide information about the location of the container you want to run the command against. To do this you just set the `container` parameter. An example use in this case would be: ``` bash qunex_container hcp_freesurfer \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1,sub2,sub3" \ --sessionsfolder="/myStudy/sessions" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` In the example above some of the parameters are related to the `qunex_container` command (e.g., `container`) while others are related to the [`hcp_freesurfer`](../../api/gmri/hcp_freesurfer.rst) command (e.g., `sessionids`, `overwrite`, etc.). QuNex automatically distinguishes these parameters and properly propagates them to relevant commands. The second possibility is to provide the information about the location of the container by setting an environmental variable `QUNEXCONIMAGE`, possibly in your `.bash_profile` or another environment script, e.g.: * For the Docker container: ``` bash export QUNEXCONIMAGE="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:" ``` * For the Apptainer container: ``` bash export QUNEXCONIMAGE="//.sif" ``` If you want to run a set of commands within a container, instead of a single command, you can make use of the [`run_turnkey`](../../api/gmri/run_turnkey.rst) QuNex commands. Please consult [QuNex quick start](../Overview/QuickStart) and [QuNex 'turnkey' workflow](../UsageDocs/Turnkey) for a detailed description of this. Another option is to write a script with commands you wish to run within the container and pass the script to the container using the `script` parameter, e.g: ``` bash qunex_container \ --script="/myStudy/processing/myScript.sh" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" ``` For more detailed information about the use of the `qunex_container` command and how to pass additional settings for the container, consult the inline help by running `qunex_container`. # Binding/mapping external folders or setting additional container parameters When using a singularity container you can bind external folders by using the `bind` parameter, this is analog to the singularity's `-B` flag. To bind multiple folders use a comma separated list: ``` bash qunex_container hcp_freesurfer \ --batchfile="/my_study/processing/batch.txt" \ --sessions="sub1,sub2,sub3" \ --sessionsfolder="/my_study/sessions" \ --bind="/my_data/data:/data,/my_data/scripts:/scripts" --container="/my_singularity_containers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` When using a docker container, QuNex automatically binds the folder you are currently in as `/data` inside the container. To change this mapping you can use the `dockeropt` parameter. The `dockeropt` is string that lists the additional options to be used when running the Docker container. If the parameter is ommited, the content of `QUNEXDOCKEROPT` environment variable will be used. If neither is specified, the container will mount the current as `data` by specifying the following options `-v "$(pwd)":/data`. The parameters are to be specified in a string exactly as they would be on a command line, e.g. to run in detached mode and mount a specific folder use ``` bash qunex_container hcp_freesurfer \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1,sub2,sub3" \ --sessionsfolder="/myStudy/sessions" \ --dockeropt="-d -v /data/host_directory/qx_study/:/my_study/" \ --container="gitlab.qunex.yale.edu:5002/qunex/qunexcontainer:" \ --overwrite="yes" ``` # Using CUDA libraries with an Apptainer container When using CUDA (GPU processing) inside an Apptainer (Singularity) container, the `--nv` flag is often required so CUDA drivers and libraries are properly setup. To achieve this, use the `--nv` flag in the `qunex_container` call: ``` bash qunex_container hcp_diffusion \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1" \ --sessionsfolder="/myStudy/sessions" \ --nv \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` Also when scheduling you have to use partitions that have GPU nodes, these partitions usually have a `_gpu` suffix in their name. If you are using Apptainer (Singularity) and a CUDA version that is not the QuNex's default, you also need to provide QuNex with the path to the location of the local CUDA libraries: ``` bash qunex_container hcp_diffusion \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1" \ --sessionsfolder="/myStudy/sessions" \ --nv \ --cuda_path="/usr/local/cuda-11/" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` ## Checking QuNex environment status inside the container You can use the `--env_status` flag to check whether everything is OK with the QuNex environment inside the container. This comes in handy when troubleshooting execution of various commands, the first thing one should check is if the QuNex environment is properly set up. ```bash qunex_container \ --env_status \ --container="/myApptainerContainers/qunexcontainer-latest.sif" ``` # Executing custom bash commands In the example above we executed the `hcp_diffusion ` command within the container by using the following command: ``` bash qunex_container hcp_diffusion \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1" \ --sessionsfolder="/myStudy/sessions" \ --nv \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` Sometimes you would like to execute some additional bash commands, either before entering the container or when already inside the container but before executing the actual QuNex command (`hcp_diffusion ` in the example above). You can do that by using the `bash_pre` and `bash_post` parameters. Commands specified in the `bash_pre` parameter will execute before entering the container while those specified in the `bash_post` parameter will execute when entering the container, but before executing the actual QuNex command. In the example below we first move to a certain directory and load the singularity module (this is done via the `bash_pre` parameter), next we enter the QuNex container, then we set the value of an environmental variable (`FSL_FIXDIR`) via the `bash_post` parameter, finally we execute the `hcp_diffusion ` command. ``` bash qunex_container hcp_diffusion \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1" \ --sessionsfolder="/myStudy/sessions" \ --nv \ --bash_pre="cd /studies/qunex/;module load singularity" \ --bash_post="export FSL_FIXDIR=/studies/qunex/fix" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` As you can see multiple bash commands can be chained together by using the semicolon character. # Executing custom bash scripts in the QuNex environment Inside the QuNex container you will find many neuroimaging tools that are fully installed and preconfigured. A common scenario is that one might need to use this tools directly for some processing or analysis since QuNex does not cover all the possible use cases for these tools. For this purpose you can use the `qunex_container` script to execute a custom bash script within the container. To achieve this you simply need to prepare a bash script that includes the desired commands and make that script executable. Next, we pass the script to `qunex_container` with the `script` parameter. For example, if I want to merge two images using `fslmerge` I would prepare this script: ``` bash #!/bin/bash flsmerge -t /data/mri_data/merged.nii.gz /data/mri_data/image1.nii.gz /data/mri_data/image2.nii.gz ``` Let's say the script is saved as `/data/mri_scripts/flsmerge.sh`. To execute it using FSL in the QuNex container I would then run: ``` bash qunex_container \ --script="/data/mri_scripts/flsmerge.sh" \ --container="/data/qx_containers/qunex_suite-0.98.0.sif" ``` If I were to add the `scheduler` parameter execution of my custom script will be scheduled on a HPC cluster. # Scheduling commands to run against the QuNex container on a cluster To make use of the parallelization in a cluster environment, `qunex_container` can schedule running the specified QuNex command as individual jobs to multiple computer cluster nodes, so that each job will process only one, or a set of sessions (or subjects) in parallel with all the others. Currently PBS and SLURM scheduling systems are supported. Scheduling is enabled by specification of additional parameters to the `qunex_container` command. First, to use a scheduler, provide a scheduler settings string to the `scheduler` parameter. The general information about the settings string is available as inline documentation by running `qunex_container`, whereas more detailed information is available by running `qunex_container ?scheduler`. An example of scheduling `qunex_container` to a single node: ``` bash qunex_container hcp_freesurfer \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1,sub2,sub3" \ --sessionsfolder="/myStudy/sessions" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --scheduler="SLURM,time=2-00:00:00,cpus-per-task=3,mem-per-cpu=20GB,partition=week" \ --overwrite="yes" ``` This command will schedule the QuNex `hcp_freesurfer` command to be run as a single job with three sessions to process. To distribute commands over multiple jobs and nodes to run in parallel, you have to provide two parameters: 1. `sessions`, and 2. `parsessions`. The first parameter provides a list of sessions (subjects) that need to be processed, the second parameter provides information on how many sessions are to be run in a single job. `qunex_container` will then schedule as many jobs as necessary to process all the sessions, so that at most N (N being equal to `parsessions`) are scheduled per job. An example of scheduling multiple jobs is: ``` bash qunex_container hcp_freesurfer \ --batchfile="/myStudy/processing/batch.txt" \ --sessionsfolder="/myStudy/sessions" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --parsessions="4" \ --scheduler="SLURM,time=2-00:00:00,cpus-per-task=4,mem-per-cpu=16GB,partition=week" \ --overwrite="yes" ``` If there are 202 sessions listed in the `/myStudy/processing/batch.txt` file, then the upper command will schedule 51 jobs, each job running the `hcp_freesurfer` command against the QuNex container. The first 50 jobs will run 4 sessions in parallel within a container. The last job will run the remaining 2 sessions in parallel within a container. For more specific information, please see the `qunex_container` inline help by running `qunex_container`. **NOTE: Passing --sessions parameter values cleanly** In the case that --sessions parameter needs to be passed to a command, but not used to split up processing into multiple container calls (e.g., in import_dicom or import_bids commands), specify the sessions to pass to the call using the --csessions parameter. The value of that parameter will be passed on as --sessions parameter to the command. ## Scheduling example ```bash qunex_container import_dicom \ --sessionsfolder="/Studies/MyStudy/sessions/" \ --csessions="ap9374" \ --masterinbox="none" \ --check="any" \ --overwrite="no" \ --container="/software/Apptainer/qunexcontainer-latest.sif" \ --scheduler="SLURM,time=0-00:20:00,cpus-per-task=1,mem-per-cpu=16GB,partition=day" ``` In the above case, only one container will be scheduled and within the container the [`import_dicom`](../../api/gmri/import_dicom.rst) command will be run with `--sessions="ap9374"`. # Making changes to the QuNex container environment This section is for more advanced users that will sometime want to make some changes to the OS and QuNex environments inside the container. For example, through described methods you can force QuNex inside container to use libraries that are located outside of the container, on your system. Doing so, you can use versions of libraries and tools (e.g. HCP Pipelines, FSL, etc.) that are different from the ones officially packaged into the container. One option is to use the previously mentioned `bash_post` parameter to execute bash code within the container. Some additional options that provide further flexibility are outlined below. ## QuNex environment Once inside the container you can run `qunex` commands. Try running `qunex` to see the help output, or `qunex_envstatus` to see the environment variables. For general QuNex usage, see [General Usage Overview](../Overview/GeneralUse). Of note, the environment script is run automatically when commands are run against the container using `qunex_container`. When run, the script (`/opt/qunex/env/qunex_environment.sh`) will reset the Apptainer shell environment and clear any prior environment variables such that it reflects the default environment state for QuNex within the container. This reset is especially important for Apptainer containers as they are "permeable" to inheriting user environment variables that were set prior to entering the Apptainer shell. There might be situations where one would like to make changes to the Apptainer shell container environment. One possibility would be to change a specific setting regarding the behavior within the container. For instance, a user can change the default options set for running octave commands, which are specified in the `QUNEXMCOMMAND` environment parameter. Another possibility is to use software and/or dependencies that are not explicitly part of the QuNex Container. For instance, a user can set an environment variable to point to a FreeSurfer installation outside of the QuNex Container default by changing the `FREESURFERDIR` environmental variable once inside the Apptainer shell. Put differently, a user can dynamically change and redeclare environment variables once inside the Apptainer shell to a use specified path. Specifically, there are three mechanisms available to change the environment variables used within the QuNex container, presented in the following three sections. ### Option I: .qunex_container.rc file If `.qunex_container.rc` file is present in the home directory, it will be sourced after the QuNex environment was reset. Notice that in the case of the Apptainer container, the home folder is the user's home folder on the host computer. In case of the Docker container the home folder is `/root` folder within the container. To have `.qunex_container.rc` present in the Docker home folder, the folder that contains the file on the host computer has to be mounted as the Docker container's `/root` folder. ### Option II: QUNEXCONTAINERENV variable If `QUNEXCONTAINERENV` is set, then it is considered to hold the path to the environment script that should be sourced within the container. The script will be sourced after the environment is reset and after checking for the presence of the `.qunex_container.rc` file. Again, there are differences between the Apptainer and the Docker container use. For Apptainer, the `QUNEXCONTAINERENV` is set in the user environment on the host computer and the path specified is a path valid on the host computer. For the Docker container, the variable has to be passed using additional Docker options, the script has to be present in a filesystem mounted to the Docker container, and the path has to be correct from within the Docker container. ### Option III: Use of con_ prefix After the environment was reset and the presence of `.qunex_container.rc` and `QUNEXCONTAINERENV` was tested, the environment script then checks for presence of valid QuNex environment variables prefixed with `con_`, e.g. `con_FREESURFERDIR` or `con_FSLDIR`. If these variables are present, the environment variables they refer to (`FREESURFERDIR`, `FSLDIR` in the previous example) will be set to their value. The formula `=${con_}` might in a specific environment result in: ``` bash export FREESURFERDIR=${con_FREESURFERDIR} export FSLDIR=${con_FSLDIR} ``` Again, note that in the case of the Apptainer container, the variables need to be set in the host computer user environment and you should not use the `--cleanenv` Apptainer flag which prevents host variables from propagating into the container. When running through `qunex_container` you should use `--nocleanenv` flag to achieve this. In the case of the Docker container, the variables need to be passed when the container is run. ## Order of operation for setting the environment The environment will be adjusted in the following order: 1. `.qunex_container.rc` 2. `QUNEXCONTAINERENV` 3. `con_` This means that if a specific variable, e.g. FSLDIR is set in `.qunex_container.rc`, and then again in the script referenced by `QUNEXCONTAINERENV`, the latter will overwrite the former. If it is again specified as `con_FSLDIR` this value will overwrite any previous value. ## Setting QuNex environment variables when using `qunex_container` Container environment variables can also be specified using the `--envars` parameter passed to `qunex_container` command. Specifically, `envars` parameter should be a pipe separated string of `"=>|=>"` pairs, illustrated below: # Example call ``` bash qunex_container hcp_freesurfer \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1,sub2,sub3" \ --sessionsfolder="/myStudy/sessions" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --envars="FSLDIR=>/usr/local/bin/fsl|FREESURFERDIR=>/opt/bin/freesurfer-6-modified" ``` Note that internally, `qunex_container` makes use of `con_*` mapping so the values specified in the `qunex_container` `--envars` parameter will have precedence over all other methods.