run_turnkey#

qx_utilities.bash.run_turnkey(sessions, sessionids, turnkeysteps, turnkeycleanstep, paramfile, mappingfile, xnathost, xnatprojectid, xnatuser, xnatpass, xnatsubjectid, xnatsubjectlabel, xnataccsessionid, xnatsessionlabel, hcp_filename, bidsname, projectname, qcplotimages, qcplotmasks, qcplotelements, turnkeytype='xnat', path='/output/xnatprojectid', acceptancetest='no', xnatstudyinputpath='input/RESOURCES/qunex_study', dataformat='DICOM', bidsformat='no', rawdatainput='', workingdir='/output', overwritestep='no', overwritesession='no', overwriteproject='no', overwriteprojectxnat='no', cleanupsession='no', cleanupproject='no', cleanupoldfiles='no', bolds='all', customqc='no')#

run_turnkey

This function implements QuNex Suite workflows as a turnkey function. It operates on a local server or cluster or within the XNAT Docker engine.

Parameters

--turnkeytype (str, default 'xnat'):

Specify type turnkey run. Options are: 'local' or 'xnat'.

--path (str, default '/output/xnatprojectid'):

Path where study folder is located. If empty default is for XNAT run.

--sessions (str):

Sessions to run locally on the file system if not an XNAT run.

--sessionids (str):

Comma separated list of session IDs to select for a run via gMRI engine from the batch file.

--turnkeysteps (str):

Specify specific turnkey steps you wish to run: Supported: TODO

--turnkeycleanstep (str):

Specify specific turnkey steps you wish to clean up intermediate files for. Supported: TODO

--paramfile (str):

File with pre-configured header specifying processing parameters.

Note: This file needs to be created manually prior to starting run_turnkey.

  • IF executing a 'local' run then provide the absolute path to the file on the local file system: If no file name is given then by default QuNex run_turnkey will exit with an error.

  • IF executing a run via the XNAT WebUI then provide the name of the file. This file should be created and uploaded manually as the project-level resource on XNAT.

--mappingfile (str):

File for mapping NIFTI files into the desired QuNex file structure (e.g. 'hcp', 'fMRIPrep' etc.)

Note: This file needs to be created manually prior to starting run_turnkey.

  • IF executing a 'local' run then provide the absolute path to the file on the local file system: If no file name is given then by default QuNex run_turnkey will exit with an error.

  • IF executing a run via the XNAT WebUI then provide the name of the file. This file should be created and uploaded manually as the project-level resource on XNAT.

--acceptancetest (str, default 'no'):

Specify if you wish to run a final acceptance test after each unit of processing.

If --acceptancetest='yes', then --turnkeysteps must be provided and will be executed first.

If --acceptancetest='<turnkey_step>', then acceptance test will be run but step won't be executed.

--xnathost (str):

Specify the XNAT site hostname URL to push data to.

--xnatprojectid (str):

Specify the XNAT site project id. This is the Project ID in XNAT and not the Project Title.

--xnatuser (str):

Specify XNAT username.

--xnatpass (str):

Specify XNAT password.

--xnatsubjectid (str):

ID for subject across the entire XNAT database. Required or --xnatsubjectlabel needs to be set.

--xnatsubjectlabel (str):

Label for subject within a project for the XNAT database. Required or --xnatsubjectid needs to be set.

--xnataccsessionid (str):

ID for subject-specific session within the XNAT project. Derived from XNAT but can be set manually.

--xnatsessionlabel (str):

Label for session within XNAT project. Note: may be general across multiple subjects (e.g. rest). Required.

--xnatstudyinputpath (str, default 'input/RESOURCES/qunex_study'):

The path to the previously generated session data as mounted for the container.

--dataformat (str, default 'DICOM'):

Specify the format in which the data is. Acceptable values are:

  • 'DICOM' ... datasets with images in DICOM format

  • 'BIDS' ... BIDS compliant datasets

  • 'HCPLS' ... HCP Life Span datasets

  • 'HCPYA' ... HCP Young Adults (1200) dataset.

--hcp_filename (str):

Specify how files and folders should be named using HCP processing:

  • 'automated' ... files should be named using QuNex automated naming (e.g. BOLD_1_PA)

  • 'userdefined' ... files should be named using user defined names (e.g. rfMRI_REST1_AP)

Note that the filename to be used has to be provided in the session_hcp.txt file or the standard naming will be used. If not provided the default 'automated' will be used.

--bidsformat (str, default 'no'):

Note: this parameter is deprecated and is kept for backward compatibility.

If set to 'yes', it will set --dataformat to BIDS. If left undefined or set to 'no', the --dataformat value will be used. The specification of the parameter follows ...

Specify if input data is in BIDS format (yes/no). Default is [no]. If set to yes, it overwrites the --dataformat parameter.

Note:

  • If --bidsformat='yes' and XNAT run is requested then --xnatsessionlabel is required.

  • If --bidsformat='yes' and XNAT run is NOT requested then BIDS data expected in <sessions_folder/inbox/BIDS.

--bidsname (str, default detailed below):

The name of the BIDS dataset. The dataset level information that does not pertain to a specific session will be stored in <projectname>/info/bids/<bidsname>. If bidsname is not provided, it will be deduced from the name of the folder in which the BIDS database is stored or from the zip package name.

--rawdatainput (str, default ''):

If --turnkeytype is not XNAT then specify location of raw data on the file system for a session. Default is '' for the XNAT type run as host is used to pull data.

--workingdir (str, default '/output'):

Specify where the study folder is to be created or resides.

--projectname (str):

Specify name of the project on local file system if XNAT is not specified.

--overwritestep (str, default 'no'):

Specify 'yes' or 'no' for delete of prior workflow step.

--overwritesession (str, default 'no'):

Specify 'yes' or 'no' for delete of prior session run.

--overwriteproject (str, default 'no'):

Specify 'yes' or 'no' for delete of entire project prior to run.

--overwriteprojectxnat (str, default 'no'):

Specify 'yes' or 'no' for delete of entire XNAT project folder prior to run.

--cleanupsession (str, default 'no'):

Specify 'yes' or 'no' for cleanup of session folder after steps are done.

--cleanupproject (str, default 'no'):

Specify 'yes' or 'no' for cleanup of entire project after steps are done.

--cleanupoldfiles (str, default 'no'):

Specify <yes> or <no> for cleanup of files that are older than start of run (XNAT run only).

--bolds (str, default 'all'):

For commands that work with BOLD images this flag specifies which specific BOLD images to process. The list of BOLDS has to be specified as a comma or pipe '|' separated string of bold numbers or bold tags as they are specified in the session_hcp.txt or batch.txt file.

Example: '--bolds=1,2,rest' would process BOLD run 1, BOLD run 2 and any other BOLD image that is tagged with the string 'rest'.

If the parameter is not specified, the default value 'all' will be used. In this scenario every BOLD image that is specified in the group batch.txt file for that session will be processed.

Note: This parameter takes precedence over the 'bolds' parameter in the batch.txt file. Therefore when run_turnkey is executed and this parameter is ommitted the '_bolds' specification in the batch.txt file never takes effect, because the default value 'all' will take precedence.

--customqc (str, default 'no'):

Either 'yes' or 'no'. If set to 'yes' then the script ooks into: ~/<study_path>/processing/scenes/QC/ for additional custom QC scenes.

Note: The provided scene has to conform to QuNex QC template standards.xw

See /opt/qunex/qx_library/data/scenes/qc/ for example templates.

The qc path has to contain relevant files for the provided scene.

--qcplotimages (str):

Absolute path to images for general_plot_bold_timeseries. See 'qunex general_plot_bold_timeseries' for help.

Only set if general_plot_bold_timeseries is requested then this is a required setting.

--qcplotmasks (str):

Absolute path to one or multiple masks to use for extracting BOLD data. See 'qunex general_plot_bold_timeseries' for help.

Only set if general_plot_bold_timeseries is requested then this is a required setting.

--qcplotelements (str):

Plot element specifications for general_plot_bold_timeseries. See 'qunex general_plot_bold_timeseries' for help.

Only set if general_plot_bold_timeseries is requested.

Notes

A complete list of commands that can be used with turnkey:

  • create_study

  • map_raw_data

  • import_dicom

  • run_qc_rawnii

  • create_session_info

  • setup_hcp

  • create_batch

  • export_hcp

  • hcp_pre_freesurfer

  • hcp_freesurfer

  • hcp_post_freesurfer

  • run_qc_t1w

  • run_qc_t2w

  • run_qc_myelin

  • hcp_fmri_volume

  • hcp_fmri_surface

  • run_qc_bold

  • hcp_diffusion

  • run_qc_dwi

  • dwi_legacy_gpu

  • dwi_eddy_qc

  • run_qc_dwi_eddy

  • dwi_dtifit

  • run_qc_dwi_dtifit

  • dwi_bedpostx_gpu

  • run_qc_dwi_process

  • run_qc_dwi_bedpostx

  • dwi_probtrackx_dense_gpu

  • dwi_pre_tractography

  • dwi_parcellate

  • dwi_seed_tractography_dense

  • run_qc_custom

  • map_hcp_data

  • create_bold_brain_masks

  • compute_bold_stats

  • create_stats_report

  • extract_nuisance_signal

  • preprocess_bold

  • preprocess_conc

  • general_plot_bold_timeseries

  • parcellate_bold

  • parcellate_bold

  • compute_bold_fc_seed

  • compute_bold_fc_gbc

  • run_qc_bold_fc.

List of Turnkey Steps:

Most turnkey steps have exact matching qunex commands with several exceptions that fall into two categories:

  • map_raw_data step is only relevant to run_turnkey, which maps files on a local filesystem or in XNAT to the study folder.

  • run_qc* and compute_bold_fc* are two groups of turnkey steps that have qunex commands as their prefixes. The suffixes of these commands are options of the corresponding qunex command.

Tracking progress:

Progress can be tracked by keeping track of the standard out and by log files generated in:

${WORK_DIR}/${STUDY_NAME}/processing/logs/runlogs/

${WORK_DIR}/${STUDY_NAME}/processing/logs/comlogs/

Examples

Run directly via:

${TOOLS}/${QUNEXREPO}/bash/qx_utilities/run_turnkey.sh \
    --<parameter1> \
    --<parameter2> \
    --<parameter3> ... \
    --<parameterN>

Run via:

qunex run_turnkey \
    --<parameter1> \
    --<parameter2> ... \
    --<parameterN>
--scheduler

A string for the cluster scheduler (e.g. LSF, PBS or SLURM) followed by relevant options.

For SLURM scheduler the string would look like this via the qunex call:

--scheduler='SLURM,jobname=<name_of_job>,time=<job_duration>,cpus-per-task=<cpu_number>,mem-per-cpu=<memory>,partition=<queue_to_send_job_to>'

qunex run_turnkey \
    --turnkeytype=<turnkey_run_type> \
    --turnkeysteps=<turnkey_worlflow_steps> \
    --paramfile=<parameters_file> \
    --overwritestep=yes \
    --mappingfile=<mapping_file> \
    --xnatsubjectlabel=<XNAT_SUBJECT_LABEL> \
    --xnatsessionlabel=<XNAT_SESSION_LABEL> \
    --xnatprojectid=<name_of_xnat_project_id> \
    --xnathostname=<XNAT_site_URL> \
    --xnatuser=<xnat_host_user_name> \
    --xnatpass=<xnat_host_user_pass>