QuNex extensions#

UNDER DEVELOPMENT

NOTE: The QuNex extensions framework is under development and not yet fully functional. This is the reason that this page is not accessible through any of the documentation menus. Before the official release, we do not promise that what is written below functions as described.

QuNex provides an integrated ecosystem for neuroimaging tasks. Some of its core features are a comprehensive logging system and the ability to run tasks via an HPC scheduler (like, e.g., SLURM). QuNex extensions make these features conveniently available to custom code.

Prerequisites

QuNex extensions are available with QuNex version 0.93.8 or later (TODO: CHECK). As usual, to run QuNex set the environment variables TOOLS and QUNEXREPO and then source the qunex_environment.sh script:

export TOOLS=/software
export QUNEXREPO=qunex
source $TOOLS/$QUNEXREPO/env/qunex_environment.sh

The qunex_environment.sh script searches for available extensions and acknowledges which ones it found.

Anatomy of a QuNex extension#

QuNex searches for extensions in three locations:

  • in $QUNEXPATH/qx_extensions where $QUNEXPATH points to the main QuNex directory

  • in $TOOLS/qx_extensions where $TOOLS can be an arbitrary directory that by default hosts QuNex installation

  • in directories that are collected in the environment variable QUNEXEXTENSIONFOLDERS

Each extension is a directory that has at least one of the following subdirectories:

  • bin

  • matlab

  • python

Bash extensions#

Bash extensions are shell scripts in the bin subfolder. These files need to be made executable and are run directly from the command line or can be referenced by other code (e.g. from Python). Note that they cannot be run via a qunex call unless their use is integrated into a python or Matlab command.

Matlab extensions#

Matlab extensions are Matlab (*.m) files in the matlab subfolder that contain a function of the same name. For example, the file matlab/hello_matlab.m must contain the Matlab function hello_matlab. Functions and scripts hosted in this folder are automatically available from within matlab environment. If there are additional subfolders that should be available within the matlab environment, these folders have to be listed in a matlabpaths file, each folder to be added to the environment on a separate line in the file. The paths provided should be relative to the extension's matlab folder.

To be able to call the functions added by the extension also from the command line using the qunex command, they must additionally be listed in a Python file together with their arguments. To do so, create a Python file in the python subdirectory, e.g. python/matlab_extensions.py. Then list the Matlab function and its arguments in a Python dictionary called functions, like so:

functions = {
    'hello_matlab': [('argument1', 'string'), ('argument2', 'string)]
}

To execute this function via QuNex run

qunex hello_matlab

Function arguments can be passed in the usual QuNex-style, e.g.

qunex hello_matlab --argument1="Hi" --argument2="Everyone"

Python Extensions#

Python extensions are searched for in the python subfolder. They can be Python (*.py) files or Python modules (i.e. subdirectories with an __init__.py file). To allow for library code that is not intended to be called directly those files and modules that should be callable from QuNex must be listed in a file called qx_modules contained in the python subfolder. For example, say there are files hello.py and world.py, as well as directories tools and lib. Then the qx_modules file

hello
tools

would expose hello.py and the module tools to qunex, but neither the file world.py nor the module lib.

QuNex will import the Python files and modules listed in qx_modules. For modules this means that the code in the module directory's __init__.py will be executed.

Simple python extensions#

To call an arbitrary Python function via QuNex, QuNex must be made aware of the function and its arguments. For instance, the function

def print_hello_fun(name=None):
    if name is None:
        print("What's your name?"
    else:
        print(f"Hello {fname}")

can be made QuNex-compatible by adding the following code to the corresponding file:

commands = {'print_hello' : { 'com': print_hello_fun, 'args': ('name',)}}

Here, print_hello is the name by which the function under com, i.e. print_hello_fun in this case, will be known to QuNex, i.e.

qunex print_hello

will execute print_hello_fun. The args entry in the dictionary must be a tuple giving the names of the function's arguments. Thus, we could run

qunex print_hello --name="Python"

Simple python extensions with decorators#

To call an arbitrary Python function via QuNex, QuNex must be made aware of the function and its arguments. Say, for instance, you have the function

def print_hello_fun(name=None):
    if name is None:
        print("What's your name?"
    else:
        print(f"Hello {fname}")

To register the function for use with QuNex requires only a few small additions:

from general.qx_ext import qx

@qx()
def print_hello(name=None):
    if name is None:
        print("What's your name?"
    else:
        print(f"Hello {fname}")

Here, @qx() is a decorator. Adding the line @qx() will leave the function print_hello unchanged, its purpose is simply to expose the function to QuNex such that it can be run as:

qunex print_hello --name="Python"

qx takes one optional argument that can be used to register a different name for the function. For example, define

from general.qx_ext import qx

@qx("say_hi")
def print_hello(name=None):
    if name is None:
        print("What's your name?"
    else:
        print(f"Hello {fname}")

Then,

qunex print_hello --name="Python"

will not work, but

qunex say_hi --name="Python"

will.

Python-based processing extensions#

(ToDo: either add explanation of the differences between simple and processing commands, or point to the relevant section in SDK once prepared.) To process sessions QuNex makes use of Python functions with a particular call signature. These functions receive information about the session to be processed when they are called and return status and logging information that qunex will then take into account. To add custom processing functions to QuNex via an extension these functions must have this particular call signature. Here's an example of such a function:

def greet_all_fun(sinfo, options, overwrite=False, thread=0):

    process

    return (r, (sinfo['id'], rstatus, report))

TODO: Document arguments sinfo, options, overwrite, thread as well as return values (r, (sinfo['id'], rstatus, report))

In addition, just like for simple Python extension function discussed above, some information about the function must be exposed in a specific way to make the compatible with qunex:

First, the function itself must be registered with QuNex. To do so, define the list calist and add an entry for the function:

calist = [
    ['ga', 'greet_all', greet_all_fun, "Greet everyone"]
]

Each such entry is itself a list with four items:

  • a short name for the command

  • a longer name for the command

  • the Python function itself

  • a description of the function

If the function processes longitudinal data add the command to lalist instead of calist. If the command processes multisession data use malist instead.

Second, as noted above, the argument options is a dictionary containing command line arguments. These command line arguments must be defined in a Python list called arglist. For example, to allow a command line argument name add the following code:

arglist = [
    ['name', '', str, 'The name to process']
]

Here, arglist contains one entry per command line argument and each such entry is a list with four entries:

  • the command line parameter itself

  • the default value to be used if the parameter is not specified

  • a Python function converting the given value of the command line parameter from a string to a different data type. In the simplest cases this can be the Python function str which effectively keeps the value a string. Other common possibilities include int, float or torf that converts 'true', 'TRUE', 'True', 'yes', ... to a boolean True or False value.

  • a description of the command line parameter

Python-based processing extensions#

To process sessions QuNex makes use of Python functions with a particular call signature. These functions receive information about the session to be processed when they are called and return status and logging information that qunex will then take into account. To add custom processing functions to QuNex via an extension these functions must have this particular call signature. Here's an example of such a function:

def greet_all(sinfo, options, overwrite=False, thread=0):

    process

    return (r, (sinfo['id'], rstatus, report))

TODO: Document arguments sinfo, options, overwrite, thread as well as return values (r, (sinfo['id'], rstatus, report))

In addition, just like for simple Python extensions discussed above, processing extensions need to be registered for use with QuNex. To do so, simply decorate the function with qx_process, i.e.:

from general.extensions import qx_process

@qx_process()
def greet_all(sinfo, options, overwrite=False, thread=0):

    process

    return (r, (sinfo['id'], rstatus, report))

If the function requires additional arguments that should be given on the command line, add these as keyword-arguments to the function definition:

@qx_process()
def greet_all(sinfo, options, overwrite=False, thread=0, name="All"):

    print(f"Hello {name}")

    return (r, (sinfo['id'], rstatus, report))

If the command processes longitudinal or multi-session data add "longitudinal" or "multisession" as an argument to qx_process, i.e. for example:

@qx_process("longitudinal")
def greet_all(sinfo, options, overwrite=False, thread=0, name="All"):

    print(f"Hello {name}")

    return (r, (sinfo['id'], rstatus, report))

Accessing the core QuNex Python library#

TODO: When I tested this I didn't need to append qx_utilities path, I could directly import functions from general package. CHECK

To use Python functions shipped with QuNex within a Python QuNex extension QUNEXPATH has to be added to the python path. This can be done by adding the following to the top of your Python file:

import os
import os.path
import sys

if "QUNEXPATH" in os.environ:
    sys.path.append(os.path.join(os.environ['QUNEXPATH'], 'python', 'qx_utilities'))
    # now QuNex Python modules are available, e.g.:
    import general.exceptions as ge
    from processing.core import *

else:
    exit(1)