Running commands over multiple sessions

Studies typically consist of multiple sessions with equal or very similar data being collected in each study. Many processing and analysis steps are therefore ran similarly for each session. QuNex puts special attention to enabling running processing and analysis over multiple sessions using a single command. This is accomplished in a number of ways, depending on the underlying command. Here we provide more detailed information on running commands over multiple sessions, depending on the specific command.

As they are aimed at accomplishing different goals with different toolsets, there are a number of different commands provided by QuNex. Some are written as bash scripts, others are implemented as python commands. Some were written by design to run over multiple sessions, others focus on specific actions that we have extended with the ability to be executed over multiple sessions.

Specifying sessions

We took special care that the user interface would be the same irrespective of the underlying code or the specific command. Three parameters answer the question of which sessions a command is to be run over, and they divide the work between them:

Parameter	What it says
--batchfile	the path to the batch file — the file the sessions and the study's parameters are read from
--sessions	which of those sessions to act on, given by id or by pattern
--filter	which of those sessions to act on, given by a <key>:<value> criterion

They mean the same thing for every command that accepts them, whether you call it directly, through a recipe, through a scheduler, or through qunex_container.

The batch file

The batch file provides a list of all — or a related subset of — the sessions in the study. It holds both detailed information about each session (paths to the session's data, the list of all its images, and arbitrary key-value descriptors such as group or gender) and the study-level parameters that apply to all of them. For details, see the specification of the batch file.

You give it as the value of --batchfile:

qunex hcp_pre_freesurfer \
    --batchfile="/data/studies/myStudy/processing/batch.txt" \
    --sessionsfolder="/data/studies/myStudy/sessions"

With no further parameters, the command runs over every session listed in the file.

Note that a batch file is more than a list of sessions. It is also where the study's parameters are stated, and QuNex reads those for every command you give the file to — including commands that act on a single session or on the study as a whole. Passing --batchfile is therefore not by itself a request to run over many sessions; it is a request to use that file's information. Which sessions are acted on is decided by the command and by the two parameters below.

Selecting sessions by id or pattern

The --sessions parameter narrows the run to some of the sessions in the batch file. It takes session ids separated by commas, spaces or pipes (|) — all three work, so pick whichever reads best:

--sessions="s12,s15,s23,s88,s92"
--sessions="ap1|ap2|ap7"

Session ids may also be given as patterns. QuNex matches them the way a shell matches file names:

Pattern	Matches
*	any number of characters
?	exactly one character
[abc]	one character out of the set

So --sessions="AP*" selects every session whose id starts with AP, and --sessions="s0?" selects s01 through s09 but not s010. Anything that is not one of those three symbols is matched literally, which means --sessions="S1" selects the session S1 and not S10.

Selecting sessions by a criterion

The --filter parameter selects sessions by the arbitrary key-value information the batch file records for each of them. It takes <key>:<value> pairs, and the values may use the same patterns as --sessions:

# every session recorded as belonging to the control group
--filter="group:control"

# either group — pairs joined by | select a session that matches any of them
--filter="group:control|group:patient"

# both criteria — pairs joined by & select a session that matches all of them
--filter="group:patient&task:rest"

Use one operator at a time: a filter that mixes | and & is reported as an error rather than guessed at.

--sessions and --filter can be combined, in which case a session has to satisfy both. This is useful when you want a criterion applied within a shortlist you have already written down:

qunex compute_bold_stats \
    --batchfile="/data/studies/myStudy/processing/batch.txt" \
    --sessions="AP*" \
    --filter="group:control" \
    --sessionsfolder="/data/studies/myStudy/sessions"

Checking a selection before you run it

Selections are easy to get subtly wrong, and a long pipeline is an expensive place to find that out. The list_sessions command prints the sessions a run would act on, and takes exactly the same three parameters:

qunex list_sessions \
    --batchfile="/data/studies/myStudy/processing/batch.txt" \
    --sessions="AP*" \
    --filter="group:control" \
    --sessionsfolder="/data/studies/myStudy/sessions"

AP1234,AP1236,AP1240

It is worth doing this whenever you write a new filter, and before scheduling anything large.

Sessions without a batch file

Some commands do their work before a batch file exists — importing data is the obvious case — or simply do not need the information a batch file holds. For those, --sessions names the sessions themselves rather than selecting within a file:

qunex import_dicom \
    --sessionsfolder="/data/studies/myStudy/sessions" \
    --sessions="or265,or319,ct17,ct21"

Patterns work here too, matched against the folders in the sessions folder, so --sessions="or*,ct*" picks up every session whose folder name starts with or or ct.

List files

When a batch file is not needed or does not yet exist, but you would rather keep a list of sessions in a file than type it out, you can use a list file. Give its path as the value of --sessions and the command runs over every session it names:

--sessions="/data/studies/myStudy/processing/pilot.list"

To learn more about list files, see the format specification for list files. They are easily created with the create_list command.

If you give both a batch file and a list file, the list file selects within the batch file, exactly as a comma-separated list of ids would.

Knowing what a command was given

Every run opens by printing the parameters the command is about to be run with, and where each value came from:

---> Parameters for hcp_pre_freesurfer

     parameter        source         value
     ---------------------------------------------------------------
     batchfile        command line   /study/processing/batch.txt
     hcp_brainsize    batch file     170
     hcp_t2           batch file     NONE
     overwrite        default        False

The same table is written into the run's log, so what a run actually used stays on record rather than having to be reconstructed afterwards. For a full account of where a parameter value can come from and which source wins, see On specification of processing parameters.

The types of commands

How you specify the sessions depends on what kind of work the command does. In general we can distinguish between the following types of commands:

• Utility commands that are designed to be run for the study as a whole and do not involve multiple sessions. An example of such a command is create_study. For these commands there is no need nor option to specify the sessions to run the command over.

• Utility commands that enable importing of data from external datasets, e.g. dicom packages from an inbox folder (import_dicom) or sessions from a BIDS dataset (import_bids). For these commands the sessions are specified using an explicit sessions list, with optional use of patterns.

• Utility commands that support aggregation of session level data. Examples are the create_batch and create_list commands. These commands can take an existing batch file through --batchfile, or an explicit sessions list or list file through --sessions.

• Processing commands that work on data before a batch file can — or needs to — be compiled. Examples are sort_dicom, dicom2niix, create_session_info and setup_hcp. These commands were designed to be executed on a single session, and QuNex runs them over each of the sessions you specify. Use --batchfile if one already exists, or name the sessions with --sessions.

• Processing and analysis commands that are designed to run over multiple sessions. These commands require a batch file as the source of both the list of sessions and the study's parameters. They are the commands that run HCP pipelines, compute additional statistics, extract and remove nuisance signal, and run GLM analyses. Examples are hcp_pre_freesurfer, hcp_fmri_volume, hcp_icafix, hcp_diffusion, compute_bold_stats, extract_nuisance_signal and preprocess_bold. For these commands the batch file is provided through --batchfile.

Whichever type you are running, --sessions and --filter narrow the set in the same way, and running the command over several sessions at once — locally or through a scheduler — is described in Parallel execution and scheduling of QuNex commands.