Running commands over multiple sessions
Running commands over multiple sessions#
Studies typically consist of multiple sessions with equal or very similar data being collected in each study. Many processing and analysis steps are therefore ran similarly for each session. QuNex puts special attention to enabling running processing and analysis over multiple sessions using a single command. This is accomplished in a number of ways, depending on the underlying command. Here we provide more detailed information on running commands over multiple sessions, depending on the specific command.
As they are aimed at accomplishing different goals with different toolsets, there are a number of different commands provided by QuNex. Some are written as bash scripts, others are implemented as python commands. Some were written by design to run over multiple sessions, others focus on specific actions that we have extended with the ability to be executed over multiple sessions.
We took special care that the user interface would be the same (or as similar as possible) irrespective of the underlying code or specific command. All the commands that can be run over multiple sessions, use the same parameters to specify the sessions to run the command over, specifically
--sessions. Depending on the command, the sessions can—or need to—be specified in different ways. These are:
Batch file provides a list of all (or a related subset of) the sessions in the study. It includes both detailed information about each session (e.g. paths to the session's data, list of all the images, arbitrary key-value descriptors for each session) as well as study level parameters. For details, see the specification of the batch file. For a number of commands, providing the batch file as the value for the
--batchfile parameter is either a necessary or an optional way to provide information on which sessions to execute the command.
When batch file is used, there are two additional parameters that can be used to limit the sessions that the command is to be run over:
--filterparameter is used to specify what criteria the sessions have to fulfill to be included in the call. It is specified as a list of key-value pairs. Only sessions that satisfy all the listed criteria are included in the call. The key-value pairs that are specified in the
--filterparameter refer to the arbitrary session specific key-value pairs listed in the batch files. The key-value pairs need to be provided as a pipe separated string, specifically in the following format:
<key>:<value>|<key>:<value>. An example of filter use is:
--filter="gender:male|rest:ok". This example would run the command only on those sessions that have in the batch file the two key-value pairs specified.
--sessionsparameter is used to explicitly state, which session (of those listed in the batch file) the command is to be run over. It is to be provided as a comma or pipe (
|) separated string of session ids. E.g.:
Sessions list string#
Some commands do not depend on the information contained in the batch file or are run even before batch file can be created (e.g.
create_batch command). For such commands it is either optional or required to provide the sessions to run the command over as an explicit list in the form of a comma or pipe (
|) separated string. An example is:
--sessions="or265,or319,or337,or365,ct17,ct21,ct33,ct52". Some commands support also grep patterns. In this case the following example could be written as
Similarly to the above case, when the information contained in the batch file is not needed or a batch file does not yet exist, the list of sessions to run the command over can be specified by a list file. In such a case, the command will be run over all the sessions listed in the list file. To learn more about list files, please see the format specification for the list files. An example in this case would be:
--sessions="/mydata/studies/mystudy/processing/pilot.list". List files can be easily created using the
The types of commands#
How to specify the sessions to run the command over, depends on the type of the command run. In general we can distinguish between the following types of commands:
Utility commands that are designed to be run for the study as a whole and do not involve multiple sessions. An example of such command is
create_study. For these commands there is no need nor option to specify the sessions to run the command over.
Utility commands that enable importing of data from external datasets, e.g. dicom packages from inbox folder (
import_dicom) or sessions from a BIDS dataset (
import_bids). For these commands the list of sessions to run the command over is specified using explicit sessions list with optional use of glob patterns.
Utility commands that support aggregation of session level data. Examples of such commands would be
create_listcommands. These commands can take as
--sessionsparameter existing batch or list files or an explicit sessions list.
Processing commands that work on data before a batch file can (or needs to) be compiled. Examples of such commands are
setup_hcp. These commands take variable parameters, specific for the task to be completed, and they were designed to be executed on a single session. To simplify running them over a number of sessions, if a
--sessionsparameter is provided, they will be automatically executed for all the specified sessions. The input to the
--sessionsparameter can be a batch file (if for some reason, it has already been created), a list file, or a sessions list string.
Processing and analysis commands that are designed to run over multiple sessions. These commands by design require batch files as a source of both the list of sessions to process as well as specific parameters. They are the commands that run HCP pipelines, compute additional statistics, extract and remove nuisance signal and run GLM analyses. Examples of such commands are
preprocess_bold, and others. For these commands the relevant batch file has to be provided as an input to