Running QuNex recipes

Often preprocessing and analysis of data in a study progresses through a number of steps using commands that are executed in a standard sequence and with a specific set of parameters. QuNex allows such sets of commands to be grouped in YAML files that contain named recipes that define command sequences. Each of these recipes can combine multiple commands with common and/or specific parameters. This setup allows one to design a predefined set of steps that can be run with a single command. This allows careful design and explicitly documented processing and analysis of all steps in a study.

The QuNex command that enables running such recipes is run_recipe. It is invoked using the following call:

qunex run_recipe \
    --recipe_file=<path to the recipe file> \
    --recipe=<name of the recipe to run> \
    [--steps=no] \
    [--logfolder=None] \
    [--verbose=no] \
    [<extra arguments>]

The run_recipe command takes two parameters:

• --recipe_file - path to a YAML file that contains recipe definitions, see below for examples.

• --recipe - the name of the recipe from the recipe_file to run.

This way the run_recipe command will execute all commands in the recipe in the defined sequence.

Logging

The log of the commands ran will be by default stored in <study>/processing/logs/runlogs stamped with date and time that the log was started. If a study folder is not yet created, please provide a valid folder to save the logs to. If the log cannot be created the run_recipe command will exit with a failure.

Individual commands that are run can generate their own logs, the presence and location of those logs depend on the specific command and settings specified in the recipe file.

Failures

run_recipe is checking for a successful completion of commands that it runs. If any of the commands fail to complete successfully, the execution of the commands will stop and the failure will be reported both in stdout as well as the log.

The recipe YAML file

The commands to run and the parameters to use when running them are specified using a recipe file in the YAML format.

Inside the YAML file, you can use the global_parameters block to define global parameters in the form of <parameter>: <value> pairs. These are the settings that will be used as defaults throughout the defined recipes and commands.

The recipes block is used for defining recipe that chain/link QuNex commands together. Each recipe starts with a name that is used for the recipe parameter of the run_recipe command, followed by a list of commands to execute. Here, you can define both recipe level parameters that are applied over all commands in a recipe, and command level parameters that are used only for a specific command.

Parameter values can also specified in the command call itself (qunex run_recipe) have the highest priority and will override all parameter values set inside the recipe file.

You can use the script keyword to invoke execution of custom bash scripts in the recipe. The script can be execute at the beginning, between QuNex commands or at the end of a recipe. Meaning that you can seamlessly intertwine your own analyses and processing in-between QuNex commands. The only thing you need to provide is the path to the script. Currently, only python and bash scripts are supported.

Example

Here is an example of a recipe file:

global_parameters:
  sessionsfolder    : /data/qx_study/sessions
  sessions          : OP101,OP102
  overwrite         : "yes"
  batchfile         : /data/qx_study/processing/batch.txt

recipes:
  onboard_dicom:
      commands:
          - script:
              path: /data/scripts/download_data.sh
          - create_study:
              studyfolder: /data/qx_study
          - import_dicom:
              masterinbox: /data/qx_data
              archive: leave
          - create_session_info:
              mapping: /data/qx_specs/hcp_mapping.txt
          - create_batch:
              targetfile: /data/qx_study/processing/batch.txt
              paramfile : /data/qx_specs/hcp_parameters.txt
          - setup_hcp

  hcp_preprocess:
      parsessions: 2

      commands:
          - hcp_pre_freesurfer
          - hcp_freesurfer
          - hcp_post_freesurfer
          - hcp_fmri_volume
          - hcp_fmri_surface

  hcp_denoise:
      commands:
          - hcp_icafix:
              hcp_matlab_mode: "{{$MATLAB_MODE}}"
          - hcp_msmall
              hcp_matlab_mode: "{{$MATLAB_MODE}}"

Examples

Here are a few examples on running lists of commands:

The first call will execute all the commands in recipe onboard_dicom. Note that this recipe starts with a script. So before executing the first QuNex command (create_study), QuNex will execute the /data/scripts/download_data.sh bash script which downloads some data that will be used later on.

qunex run_recipe \
    --recipe_file="/data/settings/recipe.yaml" \
    --recipe="onboard_dicom"

The second call will execute all the steps of the HCP minimal preprocessing pipeline in sequence. Two sessions will be processed in parallel, since we are using a scheduler, QuNex will create 1 job for each of the sessions.

qunex run_recipe \
    --recipe_file="/data/settings/recipe.yaml" \
    --recipe="hcp_preprocess"
    --batchfile="/data/qx_study/processing/batch.txt" \
    --scheduler="SLURM,jobname=hcp,time=04-00:00:00,mem-per-cpu=32G,partition=day"

The third call will execute hcp_icafix and hcp_msmall, this example showcases, how we can inject parameter values into recipes through environment variables.

export MATLAB_MODE="compiled"
qunex run_recipe \
    --recipe_file="/data/settings/recipe.yaml" \
    --recipe="hcp_denoise"

run_recipe parameters

Core parameters

run_recipe is executed using the following parameters:

• --recipe_file

  ... Path to a YAML file that contains recipe definitions.

• --recipe

  ... Name of the recipe in the recipe_file to run.

• --steps

  ... A comma separated list of steps (QuNex commands) to run. This is an alternative to specifying the recipe file and a recipe name.

• --logfolder

  ... The folder within which to save the log.

• --verbose

  ... Whether to record in a log a full verbose report of the output of each command that was run ('yes') or only a summary success report of each command that was run. ['no']

Parameter injection

Inside the recipe YAML file you can define placeholder parameter labels which can then be dynamically injected from the command call or from the system environment. To do this, encapsulate a placeholder parameter value with double curly braces:

qunex_parameter : {{$parameter_label}}

Now we can set the parameter_label by providing it as a parameter or by setting the environment variable called parameter_label, for example:

export parameter_label=<some_value>

# once the variable is set we can execute run_recipe
qunex run_recipe
  ...

Above the {parameter_label} in the recipe file will be replaced with <some_value> before the execution run_recipe.

Parameters allowing parallel processing

To setup run_recipe parallelism, you can use the traditional parsessions and parelements parameters.

• --parsessions

  ... An optional parameter specifying how many sessions to run in parallel. If parsessions parameter is already specified within the run_recipe.

• --parelements

  ... An optional parameter specifying how many elements to run in parallel within each of the jobs (e.g. how many bolds when bold processing).

• --scheduler

  ... An optional scheduler settings description string. If provided, each run_recipe invocation will be scheduled to run on a separate cluster node.

If these parameters are provided, the processing of the sessions will be split so that parsessions sessions will be processed by each separate run_recipe invocation. If scheduler is specified, each run_recipe invocation will be scheduled as a separate job on a cluster.

Please take note that if run_recipe command is ran using a scheduler, any scheduler specification within the recipe file will be ignored to avoid the attempts to spawn new cluster jobs when run_recipe instance is already running on a cluster node.