Batch files#

Preprocessing and analysis commands are typically run on batches of sessions that most often represent all the sessions in a study. Additionally, a number of parameters that are to be used in preprocessing and analysis commands are stable and do not change between command invocations. To facilitate processing of batches of sessions from small sets to thousands, QuNex utilizes batch files. The basic name for a batch file is something like batch.txt; however, any name can be used, which allows setting up separate versions of batch files for different purposes or preprocessing and analysis variants.

A batch file (e.g. batch.txt) consists of two core elements:

the parameter header
the session-level information for all sessions to be processed with the parameters in the header

The information for the parameter header is typically taken from the parameter file (commonly stored in: /<path_to_sessions_folder>/spec/parameters.txt) to allow flexible deployment and re-generation of the batch file for a given set of sessions. At the same location, you can also find parameters_example.txt which can serve as a template for preparatin of your parameter file, note that this example does not include all the possible parameters. The final batch files are by default stored in /<study_folder>/processing/ to allow easy access to all the study-specific batch iterations. In principle the final batch file can be stored anywhere as long as the relevant QuNex command specifies its absolute path.

Batch file group-level header information#

The header element of the batch file provides group-level information that will be used across all specified sessions in a form of --parameter : value pairs, one per line. Here is a concrete example of a group level section (for full details of all the parameter explanation see ~/qunex/python/qx_utilities/templates/parameters_example.txt):

# --- basic settings
--image_source:      hcp
--image_target:      cifti
--log:               keep
--glm_matrix:        both
--glm_residuals:     forget

--bold_preprocess:   WM|flanker|rest
--base_folder:       /data/studies/WM3/sessions
--TR:                2.5
--omit:              0

--pignore:           hipass=linear|regress=ignore|lopass=linear

# --- nuisance signal
--shrinknsroi:       true

# --- smoothing parameters
--surface_smooth:    4
--volume_smooth:     4
--voxel_smooth:      3
--smooth_mask:       nonzero

Note that lines starting with # are ignored and can be used for comments or to remove specific parameters.

The header is read by every command you pass the batch file to, not only by the processing commands. Each command picks up the parameters it recognises and ignores the rest, so the header is the right place for anything that holds true for the study as a whole — the image source and target, the TR, the smoothing settings, the log retention. What you type on the command line always takes precedence over what the header says, so a header value is a starting point rather than a commitment. For the full account of which level wins, see On specification of processing parameters.

Telling a command not to take what the batch file says#

Because the header reaches every command, a parameter name that two commands share is a name they now share a value for. Most of these are meant to be shared — overwrite is the same instruction whoever reads it. A few are not: targetfile is a parameter of create_batch, create_session_info, gather_behavior and pull_sequence_names, and those four write four different files. A header stating targetfile points all of them at one path.

The first answer is that whatever you type on the command line wins, so stating a value explicitly always works. When you would rather say "not from there", there are two parameters for that, one per level of the batch file:

# do not take these from the header
qunex gather_behavior --batchfile=batch.txt --unset_batch_header_parameters="targetfile"

# do not take these from any session's own entry
qunex hcp_pre_freesurfer --batchfile=batch.txt --unset_batch_session_parameters="hcp_brainsize"

Each takes a parameter name, a comma separated list of them, an array, or a glob pattern — hcp_* unsets every HCP parameter the level states. all (or *) unsets everything it states. They can be written on the command line and, in a recipe, against a command, for a recipe, or in global_parameters.

Unsetting a level is not removing a parameter. It takes that one level out of the stack and leaves the rest of it exactly as it was. Unsetting the header, which is the lowest level above the built-in defaults, means the value falls back to the default — and anything set above it, including a session's own entry and your command line, still applies over that. Unsetting a session's own entry, which is the highest level of all, means the value the levels below settled on is left standing rather than being overridden by the session.

Two things to know before using all:

It really is all — including a _sessionsfolder in the header, which is often how a command finds the study. Name the parameters you mean, or pass --sessionsfolder yourself.
The per-session level is only ever applied to processing commands, so unset_batch_session_parameters has no effect on the utility, MATLAB and Bash commands.

Every run reports what it did not take, under the parameter table, naming the parameters rather than counting them — which is how you check that a pattern removed what you meant it to.

Please note that if a parameter is specified in the batch file, QuNex will set it to the value specified. This is also the case for parameters that are left empty (e.g. _event_file :). In this case, the _event_file parameter is actually being set to an empty string and is being read as such by QuNex. The situation where this would be a misspecification would be, for instance: _nprocess :. In this case the _nprocess parameter checks how many concurrent jobs to run. If it is set to an empty string then the QuNex engine cannot set the process number. This behavior is important so that the QuNex engine can interpret all specified parameters based on actual values provided (empty or some value) as opposed to making an assumption that the parameter is absent when in fact it is present with an empty string.

Batch file session-level information following the header#

Information for each session has the same format as the session information files. To separate information from individual sessions, each new session section has to start with three dashes (---) in a line. Continuing the example from above, the session level information could start with:

---
session: NK3
subject: NK3
dicom: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK03/dicom
raw_data: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK03/nii
hpc: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK03/hpc

age: 21
handedness: right
gender: male
group: control

01: Survey
02: T1w:             T1w 0.7mm N2
03: T2w:             T2w 0.7mm N2
04: Survey
05: SE-FM-AP:        C-BOLD 3mm 48 2.5s FS-P
06: SE-FM-PA:        C-BOLD 3mm 48 2.5s FS-A
07: bold1:rest:      BOLD 3mm 48 2.5s
08: bold2:task:      BOLD 3mm 48 2.5s
09: bold2:task:      BOLD 3mm 48 2.5s

---
session: NK4
subject: NK4
dicom: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK04/dicom
raw_data: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK04/nii
hpc: /Volumes/pooh/NK/fMRI/PD-fcMRI-TMS/sessions/NK04/hpc

age: 24
handedness: left
gender: female
group: intervention

# session level parameters
--TR: 2.0

01: Survey
02: T1w:             T1w 0.7mm N2
03: T2w:             T2w 0.7mm N2
04: Survey
05: SE-FM-AP:        C-BOLD 3mm 48 2.5s FS-P
06: SE-FM-PA:        C-BOLD 3mm 48 2.5s FS-A
07: bold1:rest:      BOLD 3mm 48 2.5s
08: bold2:task:      BOLD 3mm 48 2.5s
09: bold2:task:      BOLD 3mm 48 2.5s

Note that the information above is also referenced inside the session-specific session_<pipeline>.txt files.

Generating batch files for preprocessing#

The QuNex batch files described above are the 'workhorse' of the preprocessing workflow as they can flexibly specify all relevant parameters in a header component of the file, followed by a listing of all relevant sessions to work on. Once all the data is gathered, the batch file needs to be compiled. These files (commonly named batch.txt) list information for all the sessions, as well as study parameters, so that they don't have to be specified with each invocation of a given QuNex command, which flexibly reads the relevant header parameter subset. The key advantage here is that for rapid re-processing or specification of new parameters, even within a single study, use of batch files accelerates this workflow.

To help with generation of batch files, QuNex provides a convenient create_batch command. The command scours all the subfolders in the study sessions directory for the specified session information files and joins them in the specified target file. If desired, the sessions to add can be specified explicitly or taken from another batch or list file. Run qunex create_batch --help for more detailed information on the command use.

As noted, the command also prepends the specific batch header parameters. They need to be provided if they are saved in a specific header batch parameters file. The default location of the batch header files is in /<path_to_study_folder>/sessions/specs/. The command also supports appending new sessions to an existing batch file. The final batch file with the appended session information is saved in /<path_to_study_folder>/processing/<name_of_batch_file>.txt. This file should be used in subsequent processing.

Note that if a batch file already exists then the parameter file will not be added to the header of the batch unless --overwrite parameter is set to yes. If overwrite is set to append, then the parameters will not be changed; however, any sessions that are not yet present in the batch file will be appended at the end of the batch file.

Through the batch file, you can also specify session level parameter values. These take precedence over the global parameters, and over the command line as well — they are the most specific statement about a session that the file can make. In the above example, the TR parameter is globally set to 2.5, and this value is used for processing all sessions except for NK4, where a session level TR is set to 2.0. When a session level value applies, the run reports it in that session's own parameter table, marked batch file (session).

Using a batch file in a command#

A batch file is passed to a command with the --batchfile parameter, and the sessions within it can be narrowed with --sessions and --filter:

qunex hcp_pre_freesurfer \
    --batchfile="/data/studies/WM3/processing/batch.txt" \
    --sessionsfolder="/data/studies/WM3/sessions" \
    --filter="group:control"

For the details of selecting sessions, see Running commands over multiple sessions.

Please see additional help regarding compiling a batch file on Running HCP minimal preprocessing pipeline page.

QuNex documentation

Batch files

Contents