# Running QuNex recipes Often preprocessing and analysis of data in a study progresses through a number of steps using commands that are executed in a standard sequence and with a specific set of parameters. QuNex allows such sets of commands to be grouped in YAML files that contain named recipes that define command sequences. Each of these recipes can combine multiple commands with common and/or specific parameters. This setup allows one to design a predefined set of steps that can be run with a single command. This allows careful design and explicitly documented processing and analysis of all steps in a study. The QuNex command that enables running such recipes is `run_recipe`. It is invoked using the following call: ``` bash qunex run_recipe \ --recipe_file= \ --recipe= \ [--steps=no] \ [--logfolder=None] \ [--verbose=no] \ [] ``` The `run_recipe` command takes two parameters: * `--recipe_file` - path to a YAML file that contains recipe definitions, see below for examples. * `--recipe` - the name of the recipe from the `recipe_file` to run. This way the `run_recipe` command will execute all commands in the `recipe` in the defined sequence. ## Logging The log of the commands ran will be by default stored in `/processing/logs/runlogs` stamped with date and time that the log was started. If a study folder is not yet created, please provide a valid folder to save the logs to. If the log cannot be created the `run_recipe` command will exit with a failure. Individual commands that are run can generate their own logs, the presence and location of those logs depend on the specific command and settings specified in the recipe file. ## Failures `run_recipe` is checking for a successful completion of commands that it runs. If any of the commands fail to complete successfully, the execution of the commands will stop and the failure will be reported both in stdout as well as the log. ## The recipe YAML file The commands to run and the parameters to use when running them are specified using a recipe file in the YAML format. Inside the YAML file, you can use the `global_parameters` block to define global parameters in the form of `: ` pairs. These are the settings that will be used as defaults throughout the defined recipes and commands. The `recipes` block is used for defining recipe that chain/link QuNex commands together. Each recipe starts with a name that is used for the `recipe` parameter of the `run_recipe` command, followed by a list of commands to execute. Here, you can define both recipe level parameters that are applied over all commands in a recipe, and command level parameters that are used only for a specific command. Parameter values can also specified in the command call itself (`qunex run_recipe`) have the highest priority and will override all parameter values set inside the recipe file. You can use the `script` keyword to invoke execution of custom bash scripts in the recipe. The script can be execute at the beginning, between QuNex commands or at the end of a recipe. Meaning that you can seamlessly intertwine your own analyses and processing in-between QuNex commands. The only thing you need to provide is the path to the script. Currently, only python and bash scripts are supported. ### Example Here is an example of a recipe file: ``` global_parameters: sessionsfolder : /data/qx_study/sessions sessions : OP101,OP102 overwrite : "yes" batchfile : /data/qx_study/processing/batch.txt recipes: onboard_dicom: commands: - script: path: /data/scripts/download_data.sh - create_study: studyfolder: /data/qx_study - import_dicom: masterinbox: /data/qx_data archive: leave - create_session_info: mapping: /data/qx_specs/hcp_mapping.txt - create_batch: targetfile: /data/qx_study/processing/batch.txt paramfile : /data/qx_specs/hcp_parameters.txt - setup_hcp hcp_preprocess: parsessions: 2 commands: - hcp_pre_freesurfer - hcp_freesurfer - hcp_post_freesurfer - hcp_fmri_volume - hcp_fmri_surface hcp_denoise: commands: - hcp_icafix: hcp_matlab_mode: "{{$MATLAB_MODE}}" - hcp_msmall hcp_matlab_mode: "{{$MATLAB_MODE}}" ``` ## Examples Here are a few examples on running lists of commands: The first call will execute all the commands in recipe `onboard_dicom`. Note that this recipe starts with a `script`. So before executing the first QuNex command (`create_study`), QuNex will execute the `/data/scripts/download_data.sh` bash script which downloads some data that will be used later on. ``` bash qunex run_recipe \ --recipe_file="/data/settings/recipe.yaml" \ --recipe="onboard_dicom" ``` The second call will execute all the steps of the HCP minimal preprocessing pipeline in sequence. Two sessions will be processed in parallel, since we are using a scheduler, QuNex will create 1 job for each of the sessions. ``` bash qunex run_recipe \ --recipe_file="/data/settings/recipe.yaml" \ --recipe="hcp_preprocess" --batchfile="/data/qx_study/processing/batch.txt" \ --scheduler="SLURM,jobname=hcp,time=04-00:00:00,mem-per-cpu=32G,partition=day" ``` The third call will execute `hcp_icafix` and `hcp_msmall`, this example showcases, how we can inject parameter values into recipes through environment variables. ``` bash export MATLAB_MODE="compiled" qunex run_recipe \ --recipe_file="/data/settings/recipe.yaml" \ --recipe="hcp_denoise" ``` ## run_recipe parameters ### Core parameters `run_recipe` is executed using the following parameters: * `--recipe_file` ... Path to a YAML file that contains recipe definitions. * `--recipe` ... Name of the recipe in the recipe_file to run. * `--steps` ... A comma separated list of steps (QuNex commands) to run. This is an alternative to specifying the recipe file and a recipe name. * `--logfolder` ... The folder within which to save the log. * `--verbose` ... Whether to record in a log a full verbose report of the output of each command that was run ('yes') or only a summary success report of each command that was run. ['no'] ### Parameter injection Inside the `recipe` YAML file you can define placeholder parameter labels which can then be dynamically injected from the command call or from the system environment. To do this, encapsulate a placeholder parameter value with double curly braces: ``` bash qunex_parameter : {{$parameter_label}} ``` Now we can set the `parameter_label` by providing it as a parameter or by setting the environment variable called `parameter_label`, for example: ``` bash export parameter_label= # once the variable is set we can execute run_recipe qunex run_recipe ... ``` Above the `{parameter_label}` in the recipe file will be replaced with `` before the execution `run_recipe`. ### Parameters allowing parallel processing To setup run_recipe parallelism, you can use the traditional parsessions and parelements parameters. * `--parsessions` ... An optional parameter specifying how many sessions to run in parallel. If parsessions parameter is already specified within the `run_recipe`. * `--parelements` ... An optional parameter specifying how many elements to run in parallel within each of the jobs (e.g. how many bolds when bold processing). * `--scheduler` ... An optional scheduler settings description string. If provided, each `run_recipe` invocation will be scheduled to run on a separate cluster node. If these parameters are provided, the processing of the sessions will be split so that `parsessions` sessions will be processed by each separate `run_recipe` invocation. If `scheduler` is specified, each `run_recipe` invocation will be scheduled as a separate job on a cluster. Please take note that if `run_recipe` command is ran using a scheduler, any scheduler specification within the recipe file will be ignored to avoid the attempts to spawn new cluster jobs when `run_recipe` instance is already running on a cluster node.