gather_behavior

qx_utilities.general.utilities.gather_behavior(sessionsfolder='.', batchfile=None, sessions=None, filter=None, sourcefiles='behavior.txt', targetfile=None, overwrite='no', check='yes', report='yes')

gather_behavior [sessionsfolder="."] [batchfile=None] [sessions=None] [filter=None] [sourcefiles="behavior.txt"] [targetfile="<sessionsfolder>/inbox/behavior/behavior.txt"] [overwrite="no"] [check="yes"]

Gather specified individual behavioral data from each session's behavior folder and compile it into a specified group behavioral file.

Parameters

--sessionsfolder (str, '.'):

The base study sessions folder (e.g. WM44/sessions) where the inbox and individual session folders are. If not specified, the current working folder will be taken as the location of the sessionsfolder.

--batchfile (str, None):

An optional path to a batch file or a list file to take the sessions from. sessions and filter then select within it.

--sessions (str, None):

Either a string with pipe | or comma separated list of sessions (sessions ids) to be processed; glob patterns (*, ?, [abc]) may be used, e.g. "AP128,OP139,ER*". A *.list file with a list of session ids may be given instead.

--filter (str, None):

Optional parameter used to filter sessions to include.

It is specified as a string of <key>:<value> pairs joined by | (OR) or by & (AND) — one operator at a time:

"<key>:<value>|<key>:<value>"

Only the sessions the filter selects will be included in the list. Values may be glob patterns.

--sourcefiles (str, 'behavior.txt'):

A file or comma or pipe | separated list of files or grep patterns that define, which session specific files from the behavior folder to gather data from.

--targetfile (str, None):

The path to the target file, a file that will contain the joined data from all the individual session files.

Note that targetfile is a parameter of four commands that write four different files, so a batch file header stating it points all of them at one path. State it per command, or keep it out of the header with --unset_batch_header_parameters=targetfile.

--overwrite (str, 'no'):

Whether to overwrite an existing group behavioral file or not.

--check (str, 'yes'):

Check whether all the identified sessions have data to include in the compiled group file.

The possible options are: - yes (check and report an error if no behavioral data exists for a session) - warn (warn and list the sessions for which the behavioral data was not found) - no (do not run a check, ignore sessions for which no behavioral data was found

--report (str, 'yes'):

Whether to include date when file was generated and the final report in the compiled file ('yes') or not ('no').

Notes

The command will use the sessionsfolders, sessions and filter parameters to create a list of sessions to process. For each session, the command will use the sourcefiles parameter to identify behavioral files from which to compile the data from. If no file is found for a session and the check parameter is set to yes, the command will exit with an error.

Once the files for each session are identified, the command will read all the files and compile the data into a key:value dictionary for that session. Once all the sessions are processed, a group file will be generated for all the values encountered across sessions. If any session is missing data, the missing data will be identified as 'NA'

Group data will be saved to a file specified using targetfile parameter. If no path is specified, the default location will be used:

<sessionsfolder>/inbox/behavior/behavior.txt

If a target file exists, it will be deleted and replaced, if the overwrite parameter is set to 'yes'. If the overwrite parameter is set to 'no', the command will exit with an error.

File format:

Both the individual and the resulting group data is to be stored using a tab separated value format files. Any line that starts with a hash # will be ignored. The first valid line should hold the header, specifying the names of the columns. All the following lines hold the values. Individual session files should have a single line of data. The first column of the group file will hold the session id.

In addition, if report is set to 'yes' (the default), the resulting file will start with a comment line stating the date of creation, and at the end additional comment lines will list the full report of missing files and errors encounterdd while gathering behavioral data from individual sessions.

Examples

qunex gather_behavior sessions="AP*"

The command will compile behavioral data present in behavior.txt files present in all <session id>/behavior folder that match the "AP*" glob pattern in the current folder.

The resulting file will be save in the default location:

<current folder>/inbox/behavior

If any of the identified sessions do not include data or if errors are encountered when processing the data, the command will exit with an error.

qunex gather_behavior sessionsfolder="/data/myStudy/sessions" \
        sessions="AP*|OP*" sourcefiles="*test*|*results*" \
        check="warn" overwrite="yes" report="no"

The command will find all the session folders within /data/myStudy/sessions that have a behavior subfolder. It will then look for presence of any files that match "test" or "results" glob pattern. The compiled data will be saved in the default location. If a file already exists, it will be overwritten. If any errors are encountered, the command will not throw an error, however it also won't report a successful completion of the task. The resulting file will not have information on file generation or processing report.

qunex gather_behavior sessionsfolder="/data/myStudy/sessions" \
        sessions="/data/myStudy/processing/batch.txt" \
        filter="group:controls|behavioral:yes" \
        sourcefiles="*test*|*results*" \
        targetfile="/data/myStudy/analysis/n-bridge/controls.txt" \
        check="no" overwrite="yes"

The command will read the session information from the provided batch.txt file. It will then process only those sessions that have the following lines in their description:

group: control
behavioral: yes

For those sessions it will inspect '<session id>/behavior' folder for presence of files that match either 'test' or 'results' glob pattern. The compiled data will be saved to the specified target file. If the target file exists, it will be overwritten. The command will print a full report of the processing, however, it will exit with reported success even if missing files or errors were encountered.