Running QuNex recipes#

Often preprocessing and analysis of data in a study progresses through a number of steps using commands that are executed in a standard sequence and with a specific set of parameters. QuNex allows such sets of commands to be grouped in YAML files that contain named recipes that define command sequences. Each of these recipes can combine multiple commands with common and/or specific parameters. This setup allows one to design a predefined set of steps that can be run with a single command. This allows careful design and explicitly documented processing and analysis of all steps in a study.

The QuNex command that enables running such recipes is run_recipe. It is invoked using the following call:

qunex run_recipe \
    --recipe_file=<path to the recipe file> \
    --recipe=<name of the recipe to run> \
    [--steps=no] \
    [--logfolder=None] \
    [--verbose=no] \
    [<extra arguments>]

The run_recipe command takes two parameters:

--recipe_file - path to a YAML file that contains recipe definitions, see below for examples.
--recipe - the name of the recipe from the recipe_file to run.

This way the run_recipe command will execute all commands in the recipe in the defined sequence.

Logging#

The log of the commands ran will be by default stored in <study>/processing/logs/runlogs stamped with date and time that the log was started. If a study folder is not yet created, please provide a valid folder to save the logs to. If the log cannot be created the run_recipe command will exit with a failure.

Individual commands that are run can generate their own logs, the presence and location of those logs depend on the specific command and settings specified in the recipe file.

Failures#

run_recipe is checking for a successful completion of commands that it runs. If any of the commands fail to complete successfully, the execution of the commands will stop and the failure will be reported both in stdout as well as the log.

The recipe YAML file#

The commands to run and the parameters to use when running them are specified using a recipe file in the YAML format.

Inside the YAML file, you can use the global_parameters block to define global parameters in the form of <parameter>: <value> pairs. These are the settings that will be used as defaults throughout the defined recipes and commands.

The recipes block is used for defining recipe that chain/link QuNex commands together. Each recipe starts with a name that is used for the recipe parameter of the run_recipe command, followed by a list of commands to execute. Here, you can define both recipe level parameters that are applied over all commands in a recipe, and command level parameters that are used only for a specific command.

Parameter values can also specified in the command call itself (qunex run_recipe) have the highest priority and will override all parameter values set inside the recipe file.

Parameters a step is not to inherit#

A parameter stated for the whole run is passed to every command that accepts it. When a step must not have one, say so. This is the reliable way to express it, and the recommended one — a recipe that states what its steps do not inherit reads correctly to the next person and does not depend on QuNex working anything out.

There are two ways to write it, and they mean the same thing. Use unset_parameters, which takes a parameter name, a comma separated list of them, or an array:

- create_session_info:
    mapping          : /data/specs/hcp_mapping.txt
    unset_parameters : batchfile

or write the name itself with a leading - and no value, which keeps it on the parameter's own line:

- create_session_info:
    mapping    : /data/specs/hcp_mapping.txt
    -batchfile :

Both are accepted against a single command and in a recipe's parameters block, where they apply to every command of that recipe. Neither is allowed in global_parameters: a parameter unset for every command of the run is one you need not write at all, and QuNex will tell you so rather than ignore it.

Note that an empty value does not unset a parameter. '' and None are values that commands accept — img_suffix : "" states the empty string — so removing a parameter has to be said explicitly.

unset_parameters is about what the recipe hands a step. What a batch file hands a command is a separate question with its own two parameters, described in Batch files:

to stop	write	where it can be written
what the recipe hands a step	`unset_parameters`, or `-<name>:`	against a command, or for a recipe
what the batch file's header states	`unset_batch_header_parameters`	the command line, against a command, for a recipe, or globally
what the batch file states for one session	`unset_batch_session_parameters`	the command line, against a command, for a recipe, or globally

The batch ones are allowed in global_parameters where unset_parameters is not, and the difference is not an oversight: the recipe cannot sensibly unset what it supplies itself for every command, but "no step of this run takes anything from the header" is a statement about a different file. The batch ones also accept glob patterns and all, since a header can state a great many parameters at once.

A batch file the recipe itself creates#

Recipes commonly state the study's batchfile in global_parameters and then build that file with create_batch partway through. The commands before it cannot be given a file that does not exist yet, and create_batch cannot be told to take its sessions from the file it is about to write. QuNex therefore withholds the run wide batchfile from every step up to and including the one that writes it, and says so in the log. Those steps select their sessions from the sessions folder, as they must.

This is a safety net, not a substitute for saying what you mean. Three things follow from it:

If the batch file already exists and the recipe rebuilds it, it is still withheld — so that running a recipe a second time does what running it the first time did — and a warning says so. If one of the earlier commands is meant to read the file that is there, state batchfile against that command. If none of them is, unset_parameters: batchfile says so and the warning stops.
If nothing creates the file and it does not exist, the recipe does not start, and the error names the commands that would have been given it. This is the case of a recipe file whose batch file is built by a different recipe: state the exclusion in the recipe that runs first, and it is then correct however it is invoked.
If your recipe's commands all resolve their sessions from the sessions folder, none of this applies — nothing takes a batch file, so nothing is withheld or refused.

You can use the script keyword to invoke execution of custom bash scripts in the recipe. The script can be execute at the beginning, between QuNex commands or at the end of a recipe. Meaning that you can seamlessly intertwine your own analyses and processing in-between QuNex commands. The only thing you need to provide is the path to the script. Currently, only python and bash scripts are supported.

Example#

Here is an example of a recipe file:

global_parameters:
  sessionsfolder    : /data/qx_study/sessions
  sessions          : OP101,OP102
  overwrite         : "yes"
  batchfile         : /data/qx_study/processing/batch.txt

recipes:
  onboard_dicom:
      commands:
          - script:
              path: /data/scripts/download_data.sh
          - create_study:
              studyfolder: /data/qx_study
          - import_dicom:
              masterinbox: /data/qx_data
              archive: leave
          - create_session_info:
              mapping: /data/qx_specs/hcp_mapping.txt
          - create_batch:
              targetfile: /data/qx_study/processing/batch.txt
              paramfile : /data/qx_specs/hcp_parameters.txt
          - setup_hcp

  hcp_preprocess:
      parsessions: 2

      commands:
          - hcp_pre_freesurfer
          - hcp_freesurfer
          - hcp_post_freesurfer
          - hcp_fmri_volume
          - hcp_fmri_surface

  hcp_denoise:
      commands:
          - hcp_icafix:
              hcp_matlab_mode: "{{$MATLAB_MODE}}"
          - hcp_msmall
              hcp_matlab_mode: "{{$MATLAB_MODE}}"

Examples#

Here are a few examples on running lists of commands:

The first call will execute all the commands in recipe onboard_dicom. Note that this recipe starts with a script. So before executing the first QuNex command (create_study), QuNex will execute the /data/scripts/download_data.sh bash script which downloads some data that will be used later on.

qunex run_recipe \
    --recipe_file="/data/settings/recipe.yaml" \
    --recipe="onboard_dicom"

The second call will execute all the steps of the HCP minimal preprocessing pipeline in sequence. Two sessions will be processed in parallel, since we are using a scheduler, QuNex will create 1 job for each of the sessions.

qunex run_recipe \
    --recipe_file="/data/settings/recipe.yaml" \
    --recipe="hcp_preprocess"
    --batchfile="/data/qx_study/processing/batch.txt" \
    --scheduler="SLURM,jobname=hcp,time=04-00:00:00,mem-per-cpu=32G,partition=day"

The third call will execute hcp_icafix and hcp_msmall, this example showcases, how we can inject parameter values into recipes through environment variables.

export MATLAB_MODE="compiled"
qunex run_recipe \
    --recipe_file="/data/settings/recipe.yaml" \
    --recipe="hcp_denoise"

run_recipe parameters#

Core parameters#

run_recipe is executed using the following parameters:

--recipe_file

... Path to a YAML file that contains recipe definitions.
--recipe

... Name of the recipe in the recipe_file to run.
--steps

... A comma separated list of steps (QuNex commands) to run. This is an alternative to specifying the recipe file and a recipe name.
--logfolder

... The folder within which to save the log.
--verbose

... Whether to record in a log a full verbose report of the output of each command that was run ('yes') or only a summary success report of each command that was run. ['no']

Parameter injection#

Inside the recipe YAML file you can define placeholder parameter labels which can then be dynamically injected from the command call or from the system environment. To do this, encapsulate a placeholder parameter value with double curly braces:

qunex_parameter : {{$parameter_label}}

Now we can set the parameter_label by providing it as a parameter or by setting the environment variable called parameter_label, for example:

export parameter_label=<some_value>

# once the variable is set we can execute run_recipe
qunex run_recipe
  ...

Above the {parameter_label} in the recipe file will be replaced with <some_value> before the execution run_recipe.

Parameters allowing parallel processing#

To setup run_recipe parallelism, you can use the traditional parsessions and parelements parameters.

--parsessions

... An optional parameter specifying how many sessions to run in parallel. If parsessions parameter is already specified within the run_recipe.
--parelements

... An optional parameter specifying how many elements to run in parallel within each of the jobs (e.g. how many bolds when bold processing).
--scheduler

... An optional scheduler settings description string. If provided, each run_recipe invocation will be scheduled to run on a separate cluster node.

If these parameters are provided, the processing of the sessions will be split so that parsessions sessions will be processed by each separate run_recipe invocation. If scheduler is specified, each run_recipe invocation will be scheduled as a separate job on a cluster.

Please take note that if run_recipe command is ran using a scheduler, any scheduler specification within the recipe file will be ignored to avoid the attempts to spawn new cluster jobs when run_recipe instance is already running on a cluster node.

QuNex documentation

Running QuNex recipes

Contents