# General overview `qunex_container` is an independent, self contained python3 compatible command that enables running commands against either Docker or Singularity/Apptainer QuNex container. The command is available as part of the QuNex suite. To make use of the `qunex_container`, you will have to save a copy of the command to a folder that is in your `PATH` (e.g., `/usr/local/bin` or `/usr/bin`) or add the location of the `qunex_container` to your PATH. You can refer to the [QuNex quick start](Overview-QuickStart) for details about how to download and install this script. We suggest adding the following line to your `.bashrc` or a similar shell initialization script: ``` bash export PATH=:$PATH ``` To be sure that you have the command available, you can inspect the PATH by running `echo $PATH` or the result of `which qunex_container` commands. Once you have `qunex_container` installed, you can run QuNex commands the same way you would run them inside the container or if QuNex was installed locally, with two differences. First, replace: ``` bash qunex [] ``` with ``` bash qunex_container [] ``` When running `qunex_container` you will need to provide information about the location of the container you want to run the command against. To do this you just set the `--container` parameter. An example use in this case would be: ``` bash qunex_container hcp_freesurfer \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1,sub2,sub3" \ --sessionsfolder="/myStudy/sessions" \ --overwrite="yes" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" ``` In the example above some of the parameters are related to the `qunex_container` command (e.g., `--container`) while others are related to the [`hcp_freesurfer`](../../api/gmri/hcp_freesurfer.rst) command (e.g., `--sessions`, `--overwrite`, etc.). QuNex automatically distinguishes these parameters and properly propagates them to relevant commands. ## Binding/mapping external folders or setting additional container parameters You can bind external folders by using the `--bind` parameter of the `qunex_container` script. To bind multiple folders use a comma separated list: ``` bash qunex_container hcp_freesurfer \ --batchfile="/my_study/processing/batch.txt" \ --sessions="sub1,sub2,sub3" \ --sessionsfolder="/my_study/sessions" \ --bind="/my_data/data:/data,/my_data/scripts:/scripts" --container="/my_singularity_containers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` ## Using CUDA in the container When using CUDA (GPU processing) you will have to add the `--cuda` flag to the `qunex_container` script run. This flag will sort out everything required for passing CUDA drivers into the container. For this to work, you will need to have [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit) installed on the host system. ``` bash qunex_container hcp_diffusion \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1" \ --sessionsfolder="/myStudy/sessions" \ --cuda \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` An alternative that might work with Singularity/Apptainer containers and does not require NVIDIA Container Toolkit on the host system is to use the `--nv` flag: ``` bash qunex_container hcp_diffusion \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1" \ --sessionsfolder="/myStudy/sessions" \ --nv \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` Note that, per our tests this solution unfortunately does not work very robustly on many systems. When scheduling you have to use partitions that have GPU nodes, these partitions usually have a `_gpu` suffix in their name. Sometimes, when you are using a CUDA version that is not the QuNex's default, you might also need to provide QuNex with the path to the location of the local CUDA libraries through the `--cuda_path` parameter: ``` bash qunex_container hcp_diffusion \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1" \ --sessionsfolder="/myStudy/sessions" \ --nv \ --cuda_path="/usr/local/cuda-11/" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` ## Passing additional container options To pass any additional options to the container run, you can use the `--containeropt` parameter. The `--containeropt` parameter is string that lists the additional options to be used when running the container. The parameters are to be specified in a string exactly as they would be on a command line, e.g. to run in detached mode and mount a specific folder use: ``` bash qunex_container hcp_freesurfer \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1,sub2,sub3" \ --sessionsfolder="/myStudy/sessions" \ --containeropt="-d -v /data/host_directory/qx_study/:/my_study/" \ --container="qunex/qunex_suite:" \ --overwrite="yes" ``` ## Checking QuNex environment status inside the container You can use the `--env_status` flag to check whether everything is OK with the QuNex environment inside the container. This comes in handy when troubleshooting execution of various commands, the first thing one should check is if the QuNex environment is properly set up. ```bash qunex_container \ --env_status \ --container="/myApptainerContainers/qunexcontainer-latest.sif" ``` ## Executing custom bash commands In the example above we executed the `hcp_diffusion` command within the container by using the following command: ``` bash qunex_container hcp_diffusion \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1" \ --sessionsfolder="/myStudy/sessions" \ --nv \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` Sometimes you would like to execute some additional bash commands, either before entering the container or when already inside the container but before executing the actual QuNex command (`hcp_diffusion` in the example above). You can do that by using the `--bash_pre` and `--bash_post` parameters. Commands specified in the `--bash_pre` parameter will execute before entering the container while those specified in the `--bash_post` parameter will execute when entering the container, but before executing the actual QuNex command. In the example below we first move to a certain directory and load the singularity module (this is done via the `--bash_pre` parameter), next we enter the QuNex container, then we set the value of an environmental variable (`FSL_FIXDIR`) via the `--bash_post` parameter, finally we execute the `hcp_diffusion` command. ``` bash qunex_container hcp_diffusion \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="sub1" \ --sessionsfolder="/myStudy/sessions" \ --nv \ --bash_pre="cd /studies/qunex/;module load singularity" \ --bash_post="export FSL_FIXDIR=/studies/qunex/fix" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --overwrite="yes" ``` As you can see multiple bash commands can be chained together by using the semicolon character. ## Executing custom bash scripts in the QuNex environment Inside the QuNex container you will find many neuroimaging tools that are fully installed and preconfigured. A common scenario is that one might need to use this tools directly for some processing or analysis since QuNex does not cover all the possible use cases for these tools. For this purpose you can use the `qunex_container` script to execute a custom bash script within the container. To achieve this you simply need to prepare a bash script that includes the desired commands and make that script executable. Next, we pass the script to `qunex_container` with the `--script` parameter. For example, if I want to merge two images using `fslmerge` I would prepare this script: ``` bash #!/bin/bash flsmerge -t /data/mri_data/merged.nii.gz /data/mri_data/image1.nii.gz /data/mri_data/image2.nii.gz ``` Let's say the script is saved as `/data/mri_scripts/flsmerge.sh`. To execute it using FSL in the QuNex container I would then run: ``` bash qunex_container \ --script="/data/mri_scripts/flsmerge.sh" \ --container="/data/qx_containers/qunex_suite-0.98.0.sif" ``` ## Scheduling commands to run against the QuNex container on a cluster To make use of the parallelization in a cluster environment, `qunex_container` can schedule running the specified QuNex command as individual jobs to multiple computer cluster nodes, so that each job will process only one, or a set of sessions (or subjects) in parallel with all the others. Currently PBS and SLURM scheduling systems are supported. Scheduling is enabled by specification of additional parameters to the `qunex_container` command. First, to use a scheduler, provide a scheduler settings string to the `--scheduler` parameter. An example of scheduling `hcp_freesurfer`: ``` bash qunex_container hcp_freesurfer \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="s001,s002,s003,s004" \ --sessionsfolder="/myStudy/sessions" \ --overwrite="yes" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --scheduler="SLURM,time=2-00:00:00,cpus-per-task=2,mem-per-cpu=32GB,partition=week" ``` This command will schedule the QuNex `hcp_freesurfer` command. By default, QuNex will spawn a job for each session. In the above example, QuNex will spawn 3 jobs that will be executed in parallel. If you do not want such behavior, you can use the `parjobs` parameter to define how many jobs (at most) are spawned. For example, if I set `--parjobs=1` then QuNex will spawn a single job with all 4 sessions running in it. By default the sessions in the job will be processed sequentially. You can use the `--parsessions` parameter to control that behavior: ``` bash qunex_container hcp_freesurfer \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="s001,s002,s003,s004" \ --sessionsfolder="/myStudy/sessions" \ --parjobs="2" \ --parsessions="2" \ --overwrite="yes" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --scheduler="SLURM,time=2-00:00:00,cpus-per-task=2,mem-per-cpu=32GB,partition=week" ``` Above command will spawn 2 jobs, inside each job, 2 sessions will be processed in parallel, `s001` and `s002` in the first job, `s003` and `s004` in the second job. You can also omit the `--sessions` parameter and all sessions in the `--batchfile` will be processed. For example: ``` bash qunex_container hcp_freesurfer \ --batchfile="/myStudy/processing/batch.txt" \ --sessions="s001,s002,s003,s004" \ --sessionsfolder="/myStudy/sessions" \ --parjobs="10" \ --parsessions="5" \ --overwrite="yes" \ --container="/myApptainerContainers/qunexcontainer-latest.sif" \ --scheduler="SLURM,time=2-00:00:00,cpus-per-task=2,mem-per-cpu=32GB,partition=week" ``` if there are 100 sessions listed in the `/myStudy/processing/batch.txt` file, then the upper command will schedule 10 jobs, each job running the `hcp_freesurfer` command against the QuNex container. Within each jobs, 10 sessions will be processed with 5 being run in parallel. ## Making changes to the QuNex container environment This section is for more advanced users that will sometime want to make some changes to the OS and QuNex environments inside the container. For example, through described methods you can force QuNex inside container to use libraries that are located outside of the container, on your system. Doing so, you can use versions of libraries and tools (e.g. HCP Pipelines, FSL, etc.) that are different from the ones officially packaged into the container. One option is to use the previously mentioned `bash_post` parameter to execute bash code within the container. Some additional options that provide further flexibility are outlined below. ## QuNex environment Of note, the QuNex environment is initialized automatically when commands are run against the container using `qunex_container`. When run, the script (`/opt/qunex/env/qunex_environment.sh`) will reset the shell environment and clear any prior environment variables such that it reflects the default environment state for QuNex within the container. This reset is especially important for Singularity/Apptainer containers as they are "permeable" to inheriting user environment variables that were set prior to entering the Singularity/Apptainer shell. By default, QuNex will run Singularity/Apptainer with the `--cleanenv` flag which wipes user's settings, if you want to propagate some settings, add the `--nocleanenv` to the `qunex_container` call. There might be situations where one would like to make changes to the container environment. One possibility would be to change a specific setting regarding the behavior within the container. One possibility is to use software and/or dependencies that are not explicitly part of the QuNex Container. For instance, a user can set an environment variable to point to a FreeSurfer installation outside of the QuNex Container default by changing the `FREESURFERDIR` environmental variable once inside the container. Put differently, a user can dynamically change and redeclare environment variables once inside the container shell to a use specified path. To change or set the environment variables used within the QuNex container, you can use the `--bash_post` parameter. By using the `--bash_post` parameter, we can execute custom bash commands once inside the container. For example, setting `--bash_post="export FREESURFERDIR=/local/my_freesurfer"` will overwrite the value of the `FREESURFERDIR` variable within the QuNex environment with a custom one. Sometimes the code in `--bash_post` can become very long, in these cases, you can prepare a custom script and then simply source it within the QuNex container. For example, I can have a script `/data/qunex/qunex_container_environment.sh` and in it: ``` bash export FREESURFERDIR=/local/my_freesurfer export FSLDIR=/local/my_fsl ``` By setting `--bash_post="source /data/qunex/qunex_container_environment.sh"` above code will be executed in the container.