run_qc#

qx_utilities.bash.run_qc(sessionsfolder, sessions, modality, datapath, datafile, batchfile, timestamp, dwipath, dwidata, dtifitqc, bedpostxqc, eddyqcstats, boldprefix, boldsuffix, skipframes, bolddata, boldfcinput, hcp_filename, overwrite='no', hcp_suffix='', scenetemplatefolder='${TOOLS}/${QUNEXREPO}/qx_library/data/scenes/qc', outpath='<path_to_study_sessions_folder>/QC/<input_modality_for_qc>', scenezip='yes', userscenefile='', userscenepath='', suffix='<session_id>_<timestamp>', snronly='no', boldfc='', boldfcpath='<study_folder>/sessions/<session_id>/images/functional', processcustom='no', omitdefaults='no', sourcefile='session_hcp.txt')#

run_qc

This function runs the QC preprocessing for a specified modality / processing step.

Currently Supported: ${SupportedQC}

This function is compatible with both legacy data [without T2w scans] and HCP-compliant data [with T2w scans and DWI].

Parameters

--sessionsfolder (str):

Path to study folder that contains sessions.

--sessions (str):

Comma separated list of sessions to run.

--modality (str):

Specify the modality to perform QC on. Supported: 'rawNII', 'T1w', 'T2w', 'myelin', 'BOLD', 'DWI', 'general', 'eddyQC'.

Note: If selecting 'rawNII' this function performs QC for raw NIFTI images in <sessions_folder>/<case>/nii It requires NIFTI images in <sessions_folder>/<case>/nii/ after either BIDS import of DICOM organization.

Session-specific output: <sessions_folder>/<case>/nii/slicesdir

Uses FSL's 'slicesdir' script to generate PNGs and an HTML file in the above directory.

Note: If using 'general' modality, then visualization is $TOOLS/$QUNEXREPO/qx_library/data/scenes/qc/template_general_qc.wb.scene

This will work on any input file within the session-specific data hierarchy.

--datapath (str):

Required ==> Specify path for input path relative to the <sessions_folder> if scene is 'general'.

--datafile (str):

Required ==> Specify input data file name if scene is 'general'.

--batchfile (str):

Absolute path to local batch file with pre-configured processing parameters.

Note: It can be used in combination with --sessions to select only specific cases to work on from the batch file. If --sessions is omitted, then all cases from the batch file are processed. It can also used in combination with --bolddata to select only specific BOLD runs to work on from the batch file. If --bolddata is omitted (see below), all BOLD runs in the batch file will be processed.

--overwrite (str, default 'no'):

Delete prior QC run: yes/no.

--hcp_suffix (str, default ''):

Allows user to specify session id suffix if running HCP preprocessing variants. E.g. ~/hcp/sub001 & ~/hcp/sub001-run2 ==> Here 'run2' would be specified as --hcp_suffix='-run2'

--scenetemplatefolder (str, default '${TOOLS}/${QUNEXREPO}/qx_library/data/scenes/qc'):

Specify the absolute path name of the template folder.

Note: relevant scene template data has to be in the same folder as the template scenes.

--outpath (str, default '<path_to_study_sessions_folder>/QC/<input_modality_for_qc>'):

Specify the absolute path name of the QC folder you wish the individual images and scenes saved to. If --outpath is unspecified then files are saved to: '<path_to_study_sessions_folder>/QC/<input_modality_for_qc>'.

--scenezip (str, default 'yes'):

Yes or no. Generates a ZIP file with the scene and all relevant files for Connectome Workbench visualization. Note: If scene zip set to yes, then relevant scene files will be zipped with an updated relative base folder. All paths will be relative to this base --> <path_to_study_sessions_folder>/<session_id>/hcp/<session_id> The scene zip file will be saved to: <path_for_output_file>/<session_id>.<input_modality_for_qc>.QC.wb.zip

--userscenefile (str, default ''):

User-specified scene file name. --modality info is still required to ensure correct run. Relevant data needs to be provided.

--userscenepath (str, default ''):

Path for user-specified scene and relevant data in the same location. --modality info is still required to ensure correct run.

--timestamp (str, default detailed below):

Allows user to specify unique time stamp or to parse a time stamp from QuNex bash wrapper. Current time is used if no value is provided.

--suffix (str, default '<session_id>_<timestamp>'):

Allows user to specify unique suffix or to parse a time stamp from QuNex bash wrapper.

--dwipath (str):

Specify the input path for the DWI data (may differ across studies; e.g. 'Diffusion' or 'Diffusion' or 'Diffusion_DWI_dir74_AP_b1000b2500').

--dwidata (str):

Specify the file name for DWI data (may differ across studies; e.g. 'data' or 'DWI_dir74_AP_b1000b2500_data').

--dtifitqc (str):

Specify if dtifit visual QC should be completed (e.g. 'yes' or 'no').

--bedpostxqc (str):

Specify if BedpostX visual QC should be completed (e.g. 'yes' or 'no').

--eddyqcstats (str):

Specify if EDDY QC stats should be linked into QC folder and motion report generated (e.g. 'yes' or 'no').

--boldprefix (str):

Specify the prefix file name for BOLD dtseries data (may differ across studies depending on processing; e.g. 'BOLD' or 'TASK' or 'REST'). Note: If unspecified then QC script will assume that folder names containing processed BOLDs are named numerically only (e.g. 1, 2, 3).

--boldsuffix (str):

Specify the suffix file name for BOLD dtseries data (may differ across studies depending on processing; e.g. 'Atlas' or 'MSMAll').

--skipframes (str):

Specify the number of initial frames you wish to exclude from the BOLD QC calculation.

--snronly (str, default 'no'):

Specify if you wish to compute only SNR BOLD QC calculation and skip image generation ('yes'/'no').

--bolddata (str):

Specify BOLD data numbers separated by comma or pipe. E.g. --bolddata='1,2,3,4,5'. This flag is interchangeable with --bolds or --boldruns to allow more redundancy in specification.

Note: If --bolddata is unspecified, a batch file must be provided in --batchfile or an error will be reported. If --bolddata is empty and --batchfile is provided, by default QuNex will use the information in the batch file to identify all BOLDS to process.

--boldfc (str, default ''):

Specify if you wish to compute BOLD QC for FC-type BOLD results. Supported: pscalar or pconn. Requires --boldfc='<pconn or pscalar>', --boldfcinput=<image_input>, --bolddata or --boldruns or --bolds.

--boldfcpath (str, default '<study_folder>/sessions/<session_id>/images/functional'):

Specify path for input FC data. Requires --boldfc='<pconn or pscalar>', --boldfcinput=<image_input>, --bolddata or --boldruns or --bolds.

--boldfcinput (str):

Required. If no --boldfcpath is provided then specify only data input name after bold<Number>_ which is searched for in '<sessions_folder>/<session_id>/images/functional'.

pscalar FC: Atlas_hpss_res-mVWMWB_lpss_CAB-NP-718_r_Fz_GBC.pscalar.nii
pconn FC: Atlas_hpss_res-mVWMWB_lpss_CAB-NP-718_r_Fz.pconn.nii

Requires --boldfc='<pconn or pscalar>', --boldfcinput=<image_input>, --bolddata or --boldruns or --bolds.

--processcustom (str, default 'no'):

Either 'yes' or 'no'. If set to 'yes' then the script looks into: ~/<study_path>/processing/scenes/QC/ for additional custom QC scenes.

Note: The provided scene has to conform to QuNex QC template standards.xw

See $TOOLS/$QUNEXREPO/qx_library/data/scenes/qc/ for example templates. The qc path has to contain relevant files for the provided scene.

--omitdefaults (str, default 'no'):

Either 'yes' or 'no'. If set to 'yes' then the script omits defaults.

--sourcefile (str, default 'session_hcp.txt'):

The name of the source session.txt file.

--hcp_filename (str):

Specify how files and folders should be named using HCP processing:

'automated' ... files should be named using QuNex automated naming (e.g. BOLD_1_PA)
'userdefined' ... files should be named using user defined names (e.g. rfMRI_REST1_AP)

Note that the filename to be used has to be provided in the session_hcp.txt file or the standard naming will be used. If not provided the default 'automated' will be used.

Output files

With the exception of rawNII, the function generates 3 types of outputs, which are stored within the Study in <path_to_folder_with_sessions>/QC :

.scene files that contain all relevant data loadable into Connectome Workbench
.png images that contain the output of the referenced scene file.
.zip file that contains all relevant files to download and re-generate the scene in Connectome Workbench.

Note

For BOLD data there is also an SNR txt output if specified.

Note

For raw NIFTI QC outputs are generated in: <sessions_folder>/<case>/nii/slicesdir

Notes

Raw NIFTI visual QC:

Input: requires NIFTI images in <sessionsfolder>/<case>/nii after either BIDS import of DICOM organization
Session-specific output: <sessionsfolder>/<case>/nii/slicesdir
Uses FSL’s slicesdir script to generate PNGs and an HTML file in the above directory.
This can be invoked via the qunex run_qc command.

T1w visual QC:

Input: requires T1w images and hcp_pre_freesurfer-hcp_post_freesurfer to have run successfully
Session-specific output: /<study>/sessions/<session>/QC/T1w
- T1w.QC.png image files for each session will be located in this folder
- T1w.QC.wb.scene image files also produced and located in this folder
Group outputs of all session files is specified via the optional --outpath flag.
If --outpath is unspecified then files are saved to: /<path_to_study_sessions_folder>/QC/<input_modality_for_qc>
Log Location: logs are created in /<study>/sessions/QC/T1w/qclog
- There will be error logs in this folder if QC could not be run
- Other errors can be found by looking at the QC images formed

T2w visual QC:

Input: requires T2w images and hcp_pre_freesurfer-hcp_post_freesurfer to have run successfully
Session-specific output: /<study>/sessions/<session>/QC/T2w
- T2w.QC.png image files for each session
- T2w.QC.wb.scene image files also produced
Group outputs of all session files is specified via the optional --outpath flag.
If --outpath is unspecified then files are saved to: /<path_to_study_sessions_folder>/QC/<input_modality_for_qc>
Log Location: logs are created in /<study>/sessions/QC/T2w/qclog
- There will be error logs in this folder if QC could not be run
- Other errors will need to be found by looking at the structural images

Myelin map visual QC:

Input: requires BOLD runs and hcp_pre_freesurfer-hcp_fmri_surface to have run successfully
Session-specific output: /<study>/sessions/<session>/QC/myelin
- This will produce a .myelin.QC.wb.png image and a myelin.QC.wb.scene file for each session
Group outputs of all session files is specified via the optional --outpath flag.
If --outpath is unspecified then files are saved to: /<path_to_study_sessions_folder>/QC/<input_modality_for_qc>
Log Location: logs are created in /<study>/sessions/QC/myelin/qclog
- There will be error logs in this folder if QC could not be run
- Other errors will need to be found by looking at the myelin images

BOLD visual QC:

Input: requires BOLD runs and hcp_pre_freesurfer-hcp_fmri_surface to have run successfully
Session-specific output: /<study>/sessions/<session>/QC/BOLD
- A .GStimeseries.QC.wb.png image will be produced for each BOLD in this folder
- A .GSmap.QC.wb.png image will be produced for each BOLD in this folder
- A QC.wb.scene file will be produced for each BOLD run for each session
Group outputs of all session files is specified via the optional --outpath flag.
If --outpath is unspecified then files are saved to: /<path_to_study_sessions_folder>/QC/<input_modality_for_qc>
Log Location: logs are created in /<study>/sessions/QC/BOLD/qclog
- There will be error logs in this folder if QC could not be run
- Other errors will need to be found by looking at the BOLD images
Note that IF no BOLD --bolddata is provided, then --batchfile must be provided so that the script will look for session_<pipeline>.txt info file to determine which BOLDs to run. If neither --bolddata nor --batchfile is specified, the command will return an error.

BOLD temporal Signal-to-noise (SNR):

Input: requires BOLD runs and hcp_pre_freesurfer-hcp_fmri_surface to have run successfully
Session-specific output: /<study>/sessions/<session>/QC/BOLD
- A <SNR> file will be produced for each BOLD in this folder
Group outputs of all session files is specified via the optional --outpath flag.
If --outpath is unspecified then files are saved to: /<path_to_study_sessions_folder>/QC/<input_modality_for_qc>
Log Location: logs are created in /<study>/sessions/QC/BOLD/qclog
- There will be error logs in this folder if QC could not be run
- Other errors will need to be found by looking at the BOLD images
Note that IF no BOLD --bolddata is provided in the flag the script will look for session_<pipeline>.txt info file to determine which BOLDs to run
Note that this SNR gets calculated automatically for every BOLD
If you wish to compute SNR only but omit visual QC then use this flag:
- --snronly="yes"

BOLD FC QC for scalar and pconn data:

Input: requires BOLD runs and hcp_pre_freesurfer-hcp_fmri_surface to have run successfully
Session-specific output: /<study>/sessions/<session>/QC/BOLD
- A *.png image will be produced for each BOLD in this folder either for pconn or dscalar
If --boldfcpath is unspecified then default is: /<path_to_study_sessions_folder>/sessions/<session>/images/functional
Log Location: logs are created in /<study>/sessions/QC/BOLD/qclog
- There will be error logs in this folder if QC could not be run
- Other errors will need to be found by looking at the BOLD images
Note that IF no BOLD --bolddata is provided in the flag the script will look for session_<pipeline>.txt info file to determine which BOLDs to run

DWI visual and motion QC:

Input: requires hcp_diffusion runs and hcp_pre_freesurfer-hcp_post_freesurfer to have run successfully
Session-specific output: /<study>/sessions/<session>/QC/DWI
- DWI.QC.png and DWI.QC.wb.scene files
- DWI.bedpostx.QC.png and DWI.bedpostx.QC.wb.scene files
Group outputs of all session files is specified via the optional --outpath flag.
If --outpath is unspecified then files are saved to: /<path_to_study_sessions_folder>/QC/<input_modality_for_qc>
- DWI.dtifit.QC.png and DWI.dtifit.QC.wb.scene files
Log Location: logs are created in /<study>/sessions/QC/DWI/qclog
- There will be error logs in this folder if QC could not be run
- Other errors will need to be found by looking at the myelin images

Running BOLD QC with tag selection:

run_qc allows for processing BOLD runs directly via numeric selection (e.g. 1,2,3,4) or by using the ‘tag’ specification from the ‘batch’ file. In other words, BOLD runs 1,2 could be tagged as ‘blink’ in the following example. This way, the user can filter and select which BOLDs to QC flexibly.

Examples

Run directly via:

${TOOLS}/${QUNEXREPO}/bash/qx_utilities/run_qc.sh \
    --<parameter1> \
    --<parameter2> \
    --<parameter3> ... \
    --<parameterN>

NOTE: --scheduler is not available via direct script call.

Run via:

qunex run_qc \
    --<parameter1> \
    --<parameter2> ... \
    --<parameterN>

NOTE: scheduler is available via qunex call.

--scheduler: A string for the cluster scheduler (e.g. LSF, PBS or SLURM) followed by relevant options.

For SLURM scheduler the string would look like this via the qunex call:

--scheduler='SLURM,jobname=<name_of_job>,time=<job_duration>,cpus-per-task=<cpu_number>,mem-per-cpu=<memory>,partition=<queue_to_send_job_to>'

Raw NII QC:

qunex run_qc \
    --sessionsfolder='<path_to_study_sessions_folder>' \
    --sessions='<comma_separated_list_of_cases>' \
    --modality='rawNII'

T1w QC:

qunex run_qc \
    --sessionsfolder='<path_to_study_sessions_folder>' \
    --sessions='<comma_separated_list_of_cases>' \
    --outpath='<path_for_output_file>' \
    --scenetemplatefolder='<path_for_the_template_folder>' \
    --modality='T1w' \
    --overwrite='yes'

T2w QC:

qunex run_qc \
    --sessionsfolder='<path_to_study_sessions_folder>' \
    --sessions='<comma_separated_list_of_cases>' \
    --outpath='<path_for_output_file>' \
    --scenetemplatefolder='<path_for_the_template_folder>' \
    --modality='T2w' \
    --overwrite='yes'

Myelin QC:

qunex run_qc \
    --sessionsfolder='<path_to_study_sessions_folder>' \
    --sessions='<comma_separated_list_of_cases>' \
    --outpath='<path_for_output_file>' \
    --scenetemplatefolder='<path_for_the_template_folder>' \
    --modality='myelin' \
    --overwrite='yes'

DWI QC:

qunex run_qc \
    --sessionsfolder='<path_to_study_sessions_folder>' \
    --sessions='<comma_separated_list_of_cases>' \
    --scenetemplatefolder='<path_for_the_template_folder>' \
    --modality='DWI' \
    --outpath='<path_for_output_file>' \
    --dwidata='<file_name_for_dwi_data>' \
    --dwipath='<path_for_dwi_data>' \
    --overwrite='yes'

BOLD QC (for a specific BOLD run):

qunex run_qc \
    --sessionsfolder='<path_to_study_sessions_folder>' \
    --sessions='<comma_separated_list_of_cases>' \
    --outpath='<path_for_output_file>' \
    --scenetemplatefolder='<path_for_the_template_folder>' \
    --modality='BOLD' \
    --bolddata='1' \
    --boldsuffix='Atlas' \
    --overwrite='yes'

BOLD QC (search for all available BOLD runs):

qunex run_qc \
    --sessionsfolder='<path_to_study_sessions_folder>' \
    --sessions='<comma_separated_list_of_cases>' \
    --batchfile='<path_to_batch_file>' \
    --outpath='<path_for_output_file>' \
    --scenetemplatefolder='<path_for_the_template_folder>' \
    --modality='BOLD' \
    --boldsuffix='Atlas' \
    --overwrite='yes'

BOLD temporal SNR:

qunex run_qc \
    --sessionsfolder='<path_to_study_sessions_folder>' \
    --sessions="<comma_separated_list_of_cases>" \
    --outpath='<path_for_output_file>' \
    --modality='BOLD' \
    --snronly="yes" \
    --bolddata='BOLD_#,BOLD_#' \
    --boldsuffix='Atlas' \
    --overwrite='no' \
    --scheduler='<settings for scheduler>'

BOLD FC QC [pscalar or pconn]:

qunex run_qc \
    --overwritestep='yes' \
    --sessionsfolder='<path_to_study_sessions_folder>' \
    --sessions='<comma_separated_list_of_cases>' \
    --modality='BOLD' \
    --boldfc='<pscalar_or_pconn>' \
    --boldfcinput='<data_input_for_bold_fc>' \
    --bolddata='1' \
    --overwrite='yes'

DWI visual and motion QC:

qunex run_qc \
    --sessionsfolder='<path_to_study_sessions_folder>' \
    --sessions="<comma_separated_list_of_cases>" \
    --outpath='<path_for_output_file>' \
    --templatefolder='<path_for_the_template_folder>' \
    --modality='DWI' \
    --dwidata='data' \
    --dwipath='Diffusion' \
    --dtifitqc='yes' \
    --bedpostxqc='yes' \
    --eddyqcstats='yes' \
    --overwrite='no' \
    --scheduler='<settings for scheduler>'

Running BOLD QC with tag selection:

run_qc call across several sessions using tag selection for all BOLDS:

qunex run_qc \
    --sessionsfolder='/gpfs/loomis/pi/n3/Studies/MBLab/HCPDev/OP2/sessions' \
    --batchfile='/gpfs/loomis/pi/n3/Studies/MBLab/HCPDev/OP2/processing/batch.txt' \
    --sessions='OP268_07032014,OP269_07032014,OP270_07082014' \
    --modality='BOLD' \
    --bolddata="all" \
    --boldprefix="BOLD" \
    --boldsuffix="Atlas" \
    --overwrite='yes'

run_qc call across several sessions using tag selection for specific BOLDS tagged as 'blink':

qunex run_qc --sessionsfolder='/gpfs/loomis/pi/n3/Studies/MBLab/HCPDev/OP2/sessions' \
    --batchfile='/gpfs/loomis/pi/n3/Studies/MBLab/HCPDev/OP2/processing/batch.txt' \
    --sessions='OP270_07082014,OP269_07032014' \
    --modality='BOLD' \
    --bolddata="blink" \
    --boldprefix="BOLD" \
    --boldsuffix="Atlas" \
    --overwrite='yes'

run_qc call across several sessions using numeric selection for select BOLDS:

qunex run_qc \
    --sessionsfolder='/gpfs/loomis/pi/n3/Studies/MBLab/HCPDev/OP2/sessions' \
    --batchfile='/gpfs/loomis/pi/n3/Studies/MBLab/HCPDev/OP2/processing/batch.txt' \
    --sessions='OP270_07082014,OP269_07032014' \
    --modality='BOLD' \
    --bolddata="1,6" \
    --boldprefix="BOLD" \
    --boldsuffix="Atlas" \
    --overwrite='yes'

QuNex documentation

run_qc

run_qc#