Preprocessing and denoising HCP unprocessed data#

Introduction#

In this tutorial, we will take raw unprocessed data from one of the Human Connectome Project (HCP) sessions (HCA9503576_V1_MR) and preprocess it so it will be ready for further analyses. This sessions belongs to the Human Connectome Aging (HCA) dataset, the same tutorial would also work for Human Connectome Development (HCD). For Human Connectome Young Adults (HCYA), some things would need to change as the acquitision protocol was different for that dataset.

Data onboarding#

You can get HCP unprocessed data from HCP (https://www.humanconnectome.org) resources (BALSA, S3 bucket...). The starting point for this tutorial is a .zip package named HCA9503576_V1_MR.zip that contains the unprocessed folder/data.

Before we start processing with HCP pipelines, we need to create a QuNex study and onboard the data. Before we can do that, we need to make a new directory for your study and will set some environment variables that will be used by all commands:

# make directory for your created study
mkdir /data/studies/hca_tutorial

# container path
export CONTAINER="/data/containers/qunex_suite-1.4.3.sif"

# study folder location
export STUDY_FOLDER="/data/studies/hca_tutorial"

The first environment variable is pointing to the location of the QuNex container. You can consult the QuNex QuickStart to get instructions for downloading the container and the qunex_container script that we will use for interacting with the container. The second variable is pointing to the location of the study. Both paths need to be valid for the tutorial to work, so store the files exactly as above. If you cannot access this path, tweak the path as needed, both here and everywhere else below. We are setting these variables so we will be able to use them everywhere. The alternative is to always enter the full path or value from above, but that is prone to errors and requires additional unnecessary work. Next, we will create the QuNex study:

qunex_container create_study \
  --studyfolder="${STUDY_FOLDER}" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

A few things to unpack here, qunex_container is a specialized script that allows us to interface with the QuNex container in a more user-friendly manner. Next is the command name, create_study in this case, this is the name of the command that will be executed. After that, we have command parameters in the format --<parameter_name>=<parameter_value>. --studyfolder is the path to the study folder, by using ${STUDY_FOLDER}, we are setting the parameter value to the environment variable we defined above.

The --container is needed to provide the exact location of the container, while --bind is needed to give the system within the container access to the data folder location on the disk, the value /data:/data defines that the folder /data on the system will be also accessible as /data in the container. Once you run the command, QuNex will create an empty study with all required folders. When you run a QuNex command, it will generate several logs that you can inspect in order to verify if all went well or what went wrong in the case when the command does not finish. The logs are by default stored ${STUDY_FOLDER}/logs, in some older versions the logs are under ${STUDY_FOLDER}/processing/logs. QuNex will generate two kinds of logs, runlogs are top-level summary logs, while comlogs are very detailed logs from external tools that get used. Now, we are ready to onboard the data into the study, we can to this with:

qunex_container import_hcp \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --inbox="/data/raw/hca" \
  --sessions="HCA9503576_V1_MR" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

The --sessionsfolder parameter points to the location where QuNex sessions are stored, while --inbox points to the location where the raw data archive is. The archive can be either a subfolder within that folder or a zip package in the folder. With --sessions we define which sessions to onboard, here we are specifying a single session, but you can also onboard multiple sessions with one command call by providing a comma separated list with multiple session ids.

In this case the input data are in the HCP format. In other words, folders and files are organized as they are in HCP's unprocessed data packages. With QuNex you can easily onboard other data formats as well, such as BIDS (the import_bids command) or raw DICOMs (the import_dicom command) that are output directly by the scanner.

Before we can start processing, we need to do two more things. First, we need to prepare a QuNex batch file. The QuNex batch file contains processing parameter values as well as a list of sessions in the study along with important information about the imaging data belonging to each session. This makes the batch file an important resource that assures consistent processing across batches of sessions. Code below does two things, first it downloads the parameters file and then creates the batch file.

# download the parameters file
mkdir /data/specs
cd /data/specs
wget http://jd.mblab.si/qunex/hca_parameters.txt

# create the batch file
qunex_container create_batch \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --paramfile="/data/specs/hca_parameters.txt" \
  --targetfile="${STUDY_FOLDER}/processing/batch.txt" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

The --paramfile defines the location with the file that stores parameter values. Having these parameter values in a file is not mandatory and a user can also pass parameter values when making command calls. However, some parameters are shared between commands and this approach assures that those are set to consistent values across commands. The --targetfile defines the location where we will store the batch file. If you open it after running the command, you will see that it contains parameter values on top, followed by session/imaging info. Correct values for most of the parameters can be found in the MRI acquisition protocol, JSON sidecars that come along the imaging data or can be extracted from images using various tools, such as fslhd for example.

There are three parameters in the parameter file that are not image related, the first one is --parelements, which defines the degree of parallelism within a single session, setting it to 4 means that QuNex will process 4 BOLD images in parallel when doing functional preprocessing for example. The second one is --hcp_filename, it defines how QuNex will name images, userdefined will keep original file names, setting this to automated will use a numerical naming convention (e.g., bold1, bold2, bold3...). The third parameter, --hcp_gdcoeffs points to the location of GDC (Gradient Distortion Correction) coefficients file for the MRI scanner. Since GDC files are proprietary and owned by scanner manufacturers we cannot share them with you. You can get them directly from Siemens or from your own scanner.

The next command is called setup_hcp, since we already have a prepared batch file, we only need to run it and QuNex will prepare all sessions in the batch file so they are ready for HCP processing. This means that QuNex will create the folder structure that is compliant with HCP Pipelines and put every image and other file into the location where HCP Pipelines expects it to be.

qunex_container setup_hcp \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --batchfile="${STUDY_FOLDER}/processing/batch.txt" \
  --hcp_filename="userdefined" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

Now we are ready to start processing!

Structural preprocessing#

Since we have everything setup, processing is now quite easy and because we have most of the parameters in the batch file, command calls are quite short. The three commands below execute the HCP Pipelines structural preprocessing pipeline (PreFreeSurfer, FreeSurfer, PostFreeSurfer).

qunex_container hcp_pre_freesurfer \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --batchfile="${STUDY_FOLDER}/processing/batch.txt" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

qunex_container hcp_freesurfer \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --batchfile="${STUDY_FOLDER}/processing/batch.txt" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

qunex_container hcp_post_freesurfer \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --batchfile="${STUDY_FOLDER}/processing/batch.txt" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

Functional preprocessing#

Once we successfully complete structural preprocessing, we can start with functional preprocessing. For this purpose we will use HCP's volume and surface processing. Since we set all parameters in the batch file the commands will look very similar to the structural ones above. QuNex will automatically read all relevant parameters from the batch file and add them to the outgoing call to the HCP Pipelines command.

qunex_container hcp_fmri_volume \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --batchfile="${STUDY_FOLDER}/processing/batch.txt" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

qunex_container hcp_fmri_surface \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --batchfile="${STUDY_FOLDER}/processing/batch.txt" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

Another thing QuNex will do here is it will automatically run the HCP Pipeline command for each BOLD image in the session. Since our session has 7 BOLDs (4 resting state BOLDs and 3 task BOLDs) QuNex will execute 7 HCP Pipelines command calls. Furthermore, as we have --parelements set to 4, QuNex will process 4 BOLDs in parallel, when one finishes it will automatically queue a new one. If we set --parelements to 7 it would process all 7 in parallel, note that MRI data processing is computationally very demanding and even processing 4 BOLDs in parallel is huge task, if we overload our computer with too much work it will end up working slower than if the amount of work is manageable. Processing 4 BOLDs in parallel already requires decent hardware.

Denoising#

Next, we will run denoising commands on the functional data we just processed. There are four HCP commands in this scope, ICAFix, PostFix, MSMAll, and DeDriftAndResample. Note below that with QuNex we only need to run 2 commands, per default, PostFix will be automatically executed after ICAFix and DeDriftAndResample will be automatically executed after MSMAll, which is what one usually does. By default, ICAFix will use all BOLDs and concatenate them together as fMRI_CONCAT_ALL, if you need to configure this, please consult https://qunex.readthedocs.io/en/latest/api/gmri/hcp_icafix.html.

qunex_container hcp_icafix \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --batchfile="${STUDY_FOLDER}/processing/batch.txt" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

qunex_container hcp_msmall \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --batchfile="${STUDY_FOLDER}/processing/batch.txt" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

Diffusion processing#

The next modality we will tackle is Diffusion/DWI, the command below should look familiar by now. The only new thing is the --nv, diffusion preprocessing facilitates GPUs to significantly speed up the processing, this flag will load the required NVIDIA CUDA drivers. If you do not have a GPU, you need to replace the --nv flag with --hcp_nogpu.

qunex_container hcp_diffusion \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --batchfile="${STUDY_FOLDER}/processing/batch.txt" \
  --nv \
  --bind="/data:/data" \
  --container="${CONTAINER}"

ASL#

The final modality here is arterial spin labeling (ASL), we can run preprocessing for this with:

qunex_container hcp_asl \
  --sessionsfolder="${STUDY_FOLDER}/sessions" \
  --batchfile="${STUDY_FOLDER}/processing/batch.txt" \
  --bind="/data:/data" \
  --container="${CONTAINER}"

QuNex documentation

Preprocessing and denoising HCP unprocessed data

Contents