run_recipe#

qx_utilities.general.recipe.run_recipe(recipe_file=None, recipe=None, steps=None, startwith=None, logfolder=None, eargs=None, log_settings=None)#

run_recipe [recipe_file=None] [recipe=None] [steps=None] [startwith=None] [logfolder=None] [<extra arguments>]

A command for chaining multiple commands through recipe files and recipes.

Parameters

--recipe_file (str):: Path to a YAML file that contains recipe definitions.
--recipe (str):: Name of the recipe in the recipe_file to run.
--startwith (str, default ''):: Name of the step (QuNex command) in the recipe to start with. If provided, the execution of the recipe will start with the specified step, skipping any preceding steps.
--steps (str, default ''):: A comma separated list of steps (QuNex commands) to run. This can be used to run only a subset of commands from the list or as an alternative to specifying the recipe file and a recipe name.
--logfolder (str, default ''):: The folder within which to save the log.
--eargs:: Other arguments that can be passed to the command (see Notes).

Notes

Running external scripts:

If a command in the recipe is specified as external, it will be ran as an external command. The path to the command needs to be provided in the path parameter. The command can be a binary or a script. If it is a script, the type of the script needs to be specified through the extension (e.g. .sh, .py, .R). The parameters for the external command can be provided in the same way as for QuNex commands, but they will be passed to the command as command line arguments, by default in the form of --<parameter>=<value>. User can tweak this by setting the external_parameter_prefix and external_parameter_delimiter parameters for the script. The first one defines the prefix for the parameters (default is "--"), while the second one defines the delimiter between the parameter name and value (default is "="). For example, if the prefix is set to "-" and the delimiter is set to " ", the parameters will be passed in the form of -<parameter> <value>.

Parallelism:

These parameters allow spreading processing of multiple sessions across multiple run_recipe invocations:

--batchfile: A path to a batch.txt file.
--sessions: Either a string with comma separated list of sessions (sessions ids) to be processed (use of grep patterns is possible), e.g. "OP128,OP139,ER*" or *list file with a list of session ids.
--scheduler: An optional scheduler settings description string. If provided, each run_recipe invocation will be scheduled to run on a separate cluster node.

Please take note that if run_recipe command is ran using a scheduler, any scheduler specification within the recipe_file will be ignored to avoid the attempts to spawn new cluster jobs when run_recipe instance is already running on a cluster node.

Importantly, if scheduler is specified in the run_recipe file, do bear in mind, that all the commands in the recipe will be scheduled at the same time, and not in a succession, as run_recipe can not track execution of jobs on individual cluster nodes.

To setup run_recipe parallelism, you can use the traditional parsessions and parelements parameters.

--parsessions: An optional parameter specifying how many sessions to run in parallel.
--parelements: An optional parameter specifying how many elements to run in parallel within each of the jobs (e.g. how many bolds).

The parsessions parameter defines the number of sessions that will be ran in parallel within a single run_recipe invocation. The default is 1, which means that each session will be ran in parallel within a separate job. If parsessions is set to the number of sessions, then all the sessions will be executed in sequence within a single run_recipe invocation.

Recipe file and recipes:

run_recipe takes a recipe_file and a recipe name and executes the commands defined in the recipe. The recipe_file contains commands that should be run and parameters that it should use. Alternatively, you can provide a comma separated list of commands with the steps parameter.

The log of the commands ran will be by default stored in <study>/logs/ stamped with date and time that the log was started. If a study folder is not yet created, please provide a valid folder to save the logs to. If the log can not be created the run_recipe command will exit with a failure.

run_recipe is checking for a successful completion of commands that it runs. If any of the commands fail to complete successfully, the execution of the commands will stop and the failure will be reported both in stdout as well as the log.

Individual commands that are run can generate their own logs, the presence and location of those logs depend on the specific command and settings specified in the recipe file.

Recipe files use YAML markup language. At the top of the recipe file is the global_parameters section, where the global settings are defined in the form of <parameter>: <value> pairs. These are the settings that will be used as defaults throughout all recipes and individual commands defined in the rest of the recipe file.

Recpies are defined in the recipes portion of the file where each recipeis defined by its unique name. Each recipe has two sections, the parameters and the commands. The parameters section defines the parameters and the values that are specific to that recipe. The commands section defines the commands that are specific to that recipe along with command specific parameters. All parameters are provided in the form of <parameter>:<value> pairs. Recipe level parameters have a higher priority than global parameters, while command level parameters have a higher priority than recipe level parameters. Parameters provided through the command line interface call have the highest priority, meaning that their values will override any values in recipe files.

Parameters a step is not to inherit:

A parameter stated for the whole run is passed to every command that accepts it. When a step must not have one, say so, in either of two ways. Write unset_parameters, which takes a parameter name, a comma separated list of them, or an array:

- create_session_info:
    mapping          : /data/specs/hcp_mapping.txt
    unset_parameters : batchfile

or write the name itself with a leading - and no value:

- create_session_info:
    mapping    : /data/specs/hcp_mapping.txt
    -batchfile :

Both are accepted against a step and in a recipe's parameters section, where they hold for every step of that recipe. Neither is allowed in global_parameters: a parameter unset for every command of the run is one that need not be written at all.

Note that an empty value does not unset a parameter. '' and None are values commands accept, so img_suffix : "" states the empty string rather than removing the parameter.

unset_parameters is about what the recipe hands a step. What a batch file hands a command is a separate question, answered by unset_batch_header_parameters and unset_batch_session_parameters, which may also be written against a command, for a recipe or in global_parameters -- and, unlike this one, are allowed at the global level, the batch file not being something the recipe supplies.

A batch file the recipe itself creates:

A recipe usually states the study's batchfile globally and then builds that file with create_batch partway through. The steps before it cannot be given a file that does not exist yet, and create_batch cannot be told to take its sessions from the file it is about to write, so run_recipe withholds the run wide batchfile from every step up to and including the one that writes it, and reports that it did.

If the file already exists, it is still withheld -- so that running a recipe a second time does what running it the first time did -- and a warning says so. If one of those steps is meant to read the file that is there, state batchfile against that step.

If nothing creates the file and it does not exist, the recipe does not start: no step would ever have been able to read it. A recipe whose steps all resolve their sessions from the sessions folder is unaffected, since none of them takes a batch file.

Example recipe file:

global_parameters:
    sessionsfolder    : /data/qx_study/sessions
    sessions          : OP101,OP102
    overwrite         : yes
    batchfile         : /data/qx_study/processing/batch.txt

recipes:
    onboard_dicom:
        commands:
            - external:
                path: /scripts/data_download.sh
                url: www.dummy.com/mri/data
                out_dir: /data/qx_data
                external_parameter_prefix: "-"
                external_parameter_delimiter: " "
            - create_study:
                studyfolder: /data/qx_study
            - import_dicom:
                masterinbox: /data/qx_data
                archive: leave
            - create_session_info
                mapping: /data/qx_specs/hcp_mapping.txt
            - create_batch:
                targetfile: /data/qx_study/processing/batch.txt
                paramfile : /data/qx_specs/hcp_parameters.txt
            - setup_hcp

    hcp_preprocess:
        parameters:
            parsessions: 2

        commands:
            - hcp_pre_freesurfer
            - hcp_freesurfer
            - hcp_post_freesurfer
            - hcp_fmri_volume
            - hcp_fmri_surface

    hcp_denoise:
        commands:
            - hcp_icafix:
                hcp_matlab_mode: "{{$MATLAB_MODE}}"
            - hcp_msmall
                hcp_matlab_mode: "{{$MATLAB_MODE}}"

Examples

qunex run_recipe \
--recipe_file="/data/settings/recipe.yaml" \
--recipe="onboard_dicom"

qunex run_recipe \
--recipe_file="/data/settings/recipe.yaml" \
--recipe="hcp_preprocess" \
--batchfile="/data/testStudy/processing/batch_baseline.txt" \
--scheduler="SLURM,jobname=doHCP,time=04-00:00:00,cpus-per-task=2,mem-per-cpu=40000,partition=week"

export MATLAB_MODE="interpreted"
qunex run_recipe \
--recipe_file="/data/settings/recipe.yaml" \
--recipe="hcp_denoise"

export MATLAB_MODE="interpreted"
qunex run_recipe \
--recipe_file="/data/settings/recipe.yaml" \
--recipe="hcp_denoise" \
--steps="hcp_icafix"

qunex run_recipe \
--sessionsfolder="/data/qx_study/sessions" \
--batchfile="/data/qx_study/processing/batch.txt" \
--steps="hcp_pre_freesurfer,hcp_freesurfer,hcp_post_freesurfer"

The first call will execute all the commands in recipe onboard_dicom.

The second call will execute all the steps of the HCP preprocessing pipeline via a scheduler. It will execute two sessions in parallel within the run. in sequence.

The third call will execute the hcp_denoise list where the hcp_matlab_mode parameter will be set to "interpreted" this value will be read from the system environment variable $MATLAB_MODE. This is an example of how you can inject custom values into specially marked slots (marked with "{{<label>}}") in the recipe file. Note that the labels need to be provided in the form of a string, so they need to be encapsulated with double quotes.

The fourth call is the same as the third call, except that only hcp_icafix will be executed from the hcp_denoise list.

The fifth example shows how to use the steps parameter to run a set of commands sequentially.

QuNex documentation

run_recipe

run_recipe#