schedule#

qx_utilities.general.scheduler.schedule(command=None, script=None, settings=None, replace=None, workdir=None, environment=None, output=None, bash=None, parsessions=1, parelements=1)#

schedule [command=<command string>] [script=<path to script>] settings=<settings string> [replace=<"key:value|key:value" string>] [workdir=<path to working directory>] [environment=<path to environment setup script>] [output=<string specifying how to process output>]

Schedule the provided command.

Parameters

--command (str):

The command string to be executed by the scheduler.

--script (str):

The path to a script to be executed by the scheduler.

--settings (str):

A string specifying the scheduler to be used and the additional settings for it.

Settings string should be a comma separated list of parameters. The first parameter has to be the scheduler name (PBS, SLURM or GridEngine), the rest of the parameters are key-value pairs that are to be passed as settings to the scheduler.

Example settings strings:

"SLURM,jobname=bet1,time=03-24:00:00,cpus-per-task=2,mem-per-cpu=2500,partition=week"

--replace (str):

A string of key-value pairs that specify the specific values to be imputed into the script or command.

--workdir (str):

A path to the working directory in which the command or script is to be executed.

--environment (str):

A path to a script to be executed before all other commands to set up the environment for execution.

--output (str):

A string specifying whether to return or redirect the standard output and error. See "REDIRECTING OUTPUT" for details

--bash (str):

Used if any additional commands have to be run in the compute node before the execution of the QuNex command itself. Use a semicolon separated list to chain multiple commands.

Notes

Schedules the provided command the referenced script to be run by the specified scheduler (PBS, SLURM and GridEngine are currently supported).

command, script and settings are required parameters. If the optional parameters are not specified, they will not be used.

Value substitution in command or script

If replace parameter is set, all instances of {{key}} in the command or script will be replaced with the provided value. The key/value pairs need to be separated by the pipe character, whereas key and value need to be separated by a colon. An example replacement string:

"session:AP23791|folder:/studies/WM/sessions/AP23791"

Redirecting output

If no output is specified, the job's standard output and error (stdout, stderr) are left as is and processed by the scheduler, and the result of submitting the job is printed to standard output. Output string can specify four different directives provided by "<key>:<value>" strings separated by pipe:

stdout (specifies a path to a log file that should store standard output of the submitted job.)
stderr (specified a path to a log file that should store error output of the submitted job.)
both (specifies a path to a log file that should store joint standard and error outputs of the submitted job.)
return (specifies whether standard output ('stdout'), error outpout ('stderr'), both ('both') or none ('none') should be returned as a string from the job submission call.)

Specify "return" value only when schedule is used as a function called from another python script or function to process the result.

Examples:

"stdout:processing.log"

"stdout:processing.output.log|stderr:processing.error.log"

"both:processing.log|return:true"

Do not specify error and standard outputs both using --output parameter and scheduler specific options within settings string.

Scheduler specifics:

Each of the supported scheduler systems has a somewhat different way of specifying job parameters. Please see documentation for each of the supported schedulers to provide the correct settings. Below are the information for each of the schedulers on how to specify --settings.

PBS settings:
PBS uses various flags to specify parameters. Be careful that the settings string includes only comma separated 'key=value' pairs. Scheduler will then do its best to use the right flags. Specifically:

Keys: mem, walltime, software, file, procs, pmem, feature, host, naccesspolicy, epilogue, prologue will be submitted using:
"#PBS -l <key>=<value>"
Keys: j, m, o, S, a, A, M, q, t, e, N, l will be submitted using:
"#PBS -<key> <value>"
Key: depend will be submitted using:
"#PBS -W depend=<value>"
Key: umask will be submitted using:
"#PBS -W umask=<value>"
Key: nodes is a special case. It will be submitted as:
"#PBS -l <value>"
SLURM settings:
For SLURM any provided key/value pair will be passed in the form:
"#SBATCH --<key>=<value>"
Some of the possible parameters to set are:

partition (The partition (queue) to use.)

nodes (Total number of nodes to run on.)

cpus-per-task (Number of cores per task.)

time (Maximum wall time DD-HH:MM:SS.)

constraint (Specific node architecture.)

mem-per-cpu (Memory requested per CPU in MB.)

mail-user (Email address to send notifications to.)

mail-type (On what events to send emails.)
GridEngine settings:

For GridEngine the comma separated list of settings will be unwrapped and each setting will be passed to the scheduler "as is". For example:

---scheduler="GridEngine,N example_job,l h_rt=24:00:00,pe smp 1,l mem_free=32G"

Will be converted to the following GridEngine header:

#$ -N example_job #$ -l h_rt=24:00:00 #$ -pe smp 1 #$ -l mem_free=32G

Examples

qunex schedule command="bet t1.nii.gz brain.nii.gz" \
               settings="SLURM,jobname=bet1,time=03-24:00:00,cpus-per-task=2,mem-per-cpu=2500,partition=week"

qunex schedule command="bet {{in}} {{out}}" \
               replace="in:t1.nii.gz|out:brain.nii.gz" \
               settings="SLURM,jobname=bet1,time=03-24:00:00,cpus-per-task=2,mem-per-cpu=2500,partition=week" \
               workdir="/studies/WM/sessions/AP23791/images/structural"

QuNex documentation

schedule

schedule#