Logging and log files
Contents
Logging and log files#
The QuNex suite generates a standard set of output log files that are stored in <path_to_study_folder>/processing/logs/
when a given command is run. The different kinds of logs are stored into three subfolders called runlogs
, comlogs
, and batchlogs
. Each of them contains distinct information on the processing that was run and the outputs that were generated during the execution.
Runlog files#
Runlog files are the more general log files. They are created when a QuNex command is run independently across a number of sessions. An example of this would be running steps of the HCP minimal preprocessing workflow or BOLD preprocessing. This log will list the exact command that was run and the parameters it was run with. In addition, it will list for each session the relevant information for the files and settings and report the success or failure of processing that session. Last, it will print a short summary of the success in processing each session, one session per line.
By default runlogs are stored in <study_folder_name>/processing/logs/runlogs
; however, see --logfolder
parameter for details.
Runlogs are given unique names compiled using the following specification:
Log-<command name>-<date>_<hour>.<minute>.<microsecond>.log
Comlog files#
Comlog files provide a detailed output of the processing generated by a specific command. They are generated each time a process is started. By default, comlog files are saved in the <study_folder_name>/processing/logs/comlogs
folder. However, the default location of the logs
folder can be changed using the --logfolder
parameter (see below or by running qunex -o
).
A comlog file is generated when the command has started and is named using the following specification:
tmp_<command_name>[_BOLD<N>]_<session code>_<date>_<hour>.<minute>.<microsecond>.log
(Note, [_BOLD<N>]
is added when the command is ran for an individual BOLD run. In this case <N>
stands for the BOLD number the command was run on.)
This tmp_*
file can be inspected (e.g. using tail -f <path to the comlog file>
) while the command is running to track its execution. Once the command ends, there are three possible outcomes for the final comlog file:
If the command fails, the comlog file will be renamed to start with
error
instead oftmp
and left in the folder with name:error_<command_name>[_BOLD<N>]_<session code>_<date>_<hour>.<minute>.<microsecond>.log
If the command finishes successfully and the
--log
parameter is set tokeep
(default behavior), the comlog file will be renamed to start withdone
and left in thecomlog
folder as:done_<command_name>[_BOLD<N>]_<session code>_<date>_<hour>.<minute>.<microsecond>.log
If the command finishes successfully and the
--log
parameter is set toremove
, thetmp_*
comlog file will be deleted.If the command finishes and the last file to be generated by the command is present, but the full file check has identified some missing files, the comlog file will be renamed to start with
incomplete
and left in thecomlog
folder as:incomplete_<command_name>[_BOLD<N>]_<session code>_<date>_<hour>.<minute>.<microsecond>.log
Batchlog files#
If a command was run using a scheduler to be submitted to a cluster queue, the
standard and error outputs of each of the submitted jobs will be captured and
saved into the batchlog
folder. The files will be named using the following
specification:
<batch system>_<command name>_job<job number>.<date>_<hour>.<minute>.<microsecond>.log
Depending on the specific command that was run, the content of the batchlog file might be unique or match a runlog or a comlog file.
Log specific parameters#
There are two parameters that affect logging.
logfolder#
logfolder
parameter can be used to specify the location where the runlog and comlog files are to be saved if the desired location is other than the default. If left unset (or set to an empty string), the default location for the runlogs
and comlogs
folder will be <study_folder_name>/processing/logs/
.
log#
log
parameter is used to specify what happens to comlog files after successful completion of a command (see Comlog files above). Possible values are:
remove
Comlog files are deleted after a successful completion of the command. If the command fails, the error log will be preserved.
keep
orstudy
Comlog files will be saved to the default location in
<study>/processing/logs/comlogs
that then holds all the comlog files from all the sessions. This is the default used if the--log
parameter is not specified.session
Comlog files will be saved within each session's own folder, specifically in
<sessions folder>/<session id>/logs/comlogs
. Using this option, the logs are saved along with each session, which makes it easy to find logs specific to a session and possibly archive them with the session.hcp
Comlog files will be stored in each session's own
hcp
folder, specifically in<sessions folder>/<session id>/hcp/<session id>[<hcp suffix>]/logs/comlogs
. Using this option, the log files are stored along with the results and other files related to processing using HCPpipelines.
Do note that except remove
it is possible to specify multiple options, so that the comlog files are saved in all locations that a user would find them useful. The options can be specified using a comma or pipe (|
) separated string (e.g. --log="study|session"
).
Example#
If the following command was used:
qunex preprocess_bold \
--batchfile="<studyfolder>/processing/batch.txt" \
--sessionsfolder="<studyfolder>/sessions" \
--bold_actions="shrc" \
--bold_nuisance="m,V,WM,WB,1d" \
--glm_matrix="both" \
--pignore="hipass=linear|regress=mark|lopass=linear" \
--parsessions=10 \
--bold_preprocess="rest" \
--log="study|session" \
--scheduler="SLURM,time=00-08:00:00,cpus-per-task=1,mem-per-cpu=40000,partition=<queue name>"
For example, if the batch.txt
includes 25 sessions, each with 2 BOLD runs marked as rest
, the following logs would be generated in 3 separate folders:
1.<studyfolder>/processing/logs/runlogs
would hold 4 sets of log files (one for each BOLD):
Log-preprocess_bold-<date>_<hour>.<minute>.<second>.<microsecond>.log
The first runlog file would list the scheduled jobs that
qunex
scheduled to run on the cluster SLURM queuing system. One job for the first 10 sessions, the next for the next 10 sessions, and the last for the remaining 5 sessions.This workflow breakdown is controlled by the
--parsessions=10
parameter. If this parameter was set to 5, then there would be 5 logs (5 sessions per job).Each of the jobs scheduled would generate corresponding runlog files. Each of them would list the settings, the detailed report on the command that was executed and a summary report for each of the sessions and bolds that were part of the job.
2.<studyfolder>/processing/logs/comlogs
would hold 50 log files named:
done_Preprocess_B<N>_<session code>_<date>_<hour>.<minute>.<microsecond>.log
Here each comlog file would correspond to each of the BOLD files for each session (i.e. 2 x 25 = 50) that were part of the job. While the job is ongoing, the comlog file would start with
tmp
instead. If any of the commands fail, the related comlog file would start witherror
instead.As both
study
andsession
were specified, the same logs would also be saved in each session's folder, specifically in<sessionsfolder>/<sessionid>/logs/comlogs
.
3.<studyfolder>/processing/logs/batchlogs
would hold 3 log files named:
SLURM_preprocessBOLD_job<job number>.<date>_<hour>.<minute>.<microsecond>.log
Each of them would store the combined standard and error outputs of the 3 jobs submitted to the queuing system (in this case SLURM).
Here the content of these files would match the respective runlog files, as the qunex command generates the same standard and error output as it stores in the runlog files.