# Logging and log files The QuNex suite generates a standard set of output log files that are stored in `/processing/logs/` when a given command is run. The different kinds of logs are stored into three subfolders called `runlogs`, `comlogs`, and `batchlogs`. Each of them contains distinct information on the processing that was run and the outputs that were generated during the execution. ## Runlog files *Runlog* files are the more general log files. They are created when a QuNex command is run independently across a number of sessions. An example of this would be running steps of the HCP minimal preprocessing workflow or BOLD preprocessing. This log will list the exact command that was run and the parameters it was run with. In addition, it will list for each session the relevant information for the files and settings and report the success or failure of processing that session. Last, it will print a short summary of the success in processing each session, one session per line. By default *runlogs* are stored in `/processing/logs/runlogs`; however, see `--logfolder` parameter for details. *Runlogs* are given unique names compiled using the following specification: ```Log--_...log``` ## Comlog files *Comlog* files provide a detailed output of the processing generated by a specific command. They are generated each time a process is started. By default, *comlog* files are saved in the `/processing/logs/comlogs` folder. However, the default location of the `logs` folder can be changed using the `--logfolder` parameter (see below or by running `qunex -o`). A *comlog* file is generated when the command has started and is named using the following specification: ```tmp_[_BOLD]___...log``` (Note, `[_BOLD]` is added when the command is ran for an individual BOLD run. In this case `` stands for the BOLD number the command was run on.) This `tmp_*` file can be inspected (e.g. using `tail -f `) while the command is running to track its execution. Once the command ends, there are three possible outcomes for the final *comlog* file: 1. If the command fails, the *comlog* file will be renamed to start with `error` instead of `tmp` and left in the folder with name: ```error_[_BOLD]___...log``` 2. If the command finishes successfully and the `--log` parameter is set to `keep` (default behavior), the *comlog* file will be renamed to start with `done` and left in the `comlog` folder as: ```done_[_BOLD]___...log``` 3. If the command finishes successfully and the `--log` parameter is set to `remove`, the `tmp_*` *comlog* file will be deleted. 4. If the command finishes and the last file to be generated by the command is present, but the full file check has identified some missing files, the *comlog* file will be renamed to start with `incomplete` and left in the `comlog` folder as: ```incomplete_[_BOLD]___...log``` ## Batchlog files If a command was run using a scheduler to be submitted to a cluster queue, the standard and error outputs of each of the submitted jobs will be captured and saved into the `batchlog` folder. The files will be named using the following specification: ```__job._...log``` Depending on the specific command that was run, the content of the *batchlog* file might be unique or match a *runlog* or a *comlog* file. ## Log specific parameters There are two parameters that affect logging. ### logfolder `logfolder` parameter can be used to specify the location where the *runlog* and *comlog* files are to be saved if the desired location is other than the default. If left unset (or set to an empty string), the default location for the `runlogs` and `comlogs` folder will be `/processing/logs/`. ### log `log` parameter is used to specify what happens to *comlog* files after successful completion of a command (see *Comlog files* above). Possible values are: * `remove` Comlog files are deleted after a successful completion of the command. If the command fails, the error log will be preserved. * `keep` or `study` Comlog files will be saved to the default location in `/processing/logs/comlogs` that then holds all the comlog files from all the sessions. This is the default used if the `--log` parameter is not specified. * `session` Comlog files will be saved within each session's own folder, specifically in `//logs/comlogs`. Using this option, the logs are saved along with each session, which makes it easy to find logs specific to a session and possibly archive them with the session. * `hcp` Comlog files will be stored in each session's own `hcp` folder, specifically in `//hcp/[]/logs/comlogs`. Using this option, the log files are stored along with the results and other files related to processing using HCPpipelines. Do note that except `remove` it is possible to specify multiple options, so that the comlog files are saved in all locations that a user would find them useful. The options can be specified using a comma or pipe (`|`) separated string (e.g. `--log="study|session"`). ## Example If the following command was used: ``` bash qunex preprocess_bold \ --batchfile="/processing/batch.txt" \ --sessionsfolder="/sessions" \ --bold_actions="shrc" \ --bold_nuisance="m,V,WM,WB,1d" \ --glm_matrix="both" \ --pignore="hipass=linear|regress=mark|lopass=linear" \ --parsessions=10 \ --bold_preprocess="rest" \ --log="study|session" \ --scheduler="SLURM,time=00-08:00:00,cpus-per-task=1,mem-per-cpu=40000,partition=" ``` For example, if the `batch.txt` includes 25 sessions, each with 2 BOLD runs marked as `rest`, the following logs would be generated in 3 separate folders: 1.`/processing/logs/runlogs` would hold 4 sets of log files (one for each BOLD): ```Log-preprocess_bold-_....log``` * The first *runlog* file would list the scheduled jobs that `qunex` scheduled to run on the cluster SLURM queuing system. One job for the first 10 sessions, the next for the next 10 sessions, and the last for the remaining 5 sessions. * This workflow breakdown is controlled by the `--parsessions=10` parameter. If this parameter was set to 5, then there would be 5 logs (5 sessions per job). * Each of the jobs scheduled would generate corresponding *runlog* files. Each of them would list the settings, the detailed report on the command that was executed and a summary report for each of the sessions and bolds that were part of the job. 2.`/processing/logs/comlogs` would hold 50 log files named: ```done_Preprocess_B___...log``` * Here each *comlog* file would correspond to each of the BOLD files for each session (i.e. 2 x 25 = 50) that were part of the job. While the job is ongoing, the *comlog* file would start with `tmp` instead. If any of the commands fail, the related *comlog* file would start with `error` instead. * As both `study` and `session` were specified, the same logs would also be saved in each session's folder, specifically in `//logs/comlogs`. 3.`/processing/logs/batchlogs` would hold 3 log files named: ```SLURM_preprocessBOLD_job._...log``` * Each of them would store the combined standard and error outputs of the 3 jobs submitted to the queuing system (in this case SLURM). * Here the content of these files would match the respective *runlog* files, as the qunex command generates the same standard and error output as it stores in the *runlog* files.