This module proposes a simple scheduler used to execute a list of tasks controlling the machine load.

hopla.scheduler.scheduler(commands, outputdir=None, cpus=1, logfile=None, cluster=False, cluster_logdir=None, cluster_queue=None, cluster_memory=1, cluster_walltime=24, cluster_python_cmd='python', verbose=1)

Execute some commands (python scripts) using a scheduler.

If the script contains a ‘__hopla__’ list of parameter names to keep trace on, all the specified parameters values are stored in the execution status.

Parameters:

commands: list of list of str (mandatory)

some commands to be executed: the first command element must be a path to a python script.

outputdir: str (optional, default None)

a folder where a summary of the executed jobs are written.

cpus: int (optional, default 1)

the number of cpus to be used.

logfile: str (optional, default None)

location where the log messages are redirected: INFO and DEBUG.

cluster: bool (optional, default False)

if True use a worker that submits the jobs to a cluster.

cluster_logdir: str (optional, default None)

an existing path where the cluster error and output files will be stored. This folder must be empty.

cluster_queue: str (optional, default None)

the name of the queue where the jobs will be submited.

cluster_memory: float (optional, default 1)

the memory allocated to each job submitted on a cluster (in GB).

cluster_walltime: int (optional, default 24)

the walltime used for each job submitted on the cluster (in hours).

cluster_python_cmd: str (optional, default ‘python’)

the path to the python binary.

verbose: int (optional, default 1)

0 - display no log in console, 1 - display information log in console, 2 - display debug log in console.

Returns:

execution_status: dict

a dictionary that contains all the executed command return codes.

exitcodes: dict

a dictionary with a summary of the executed jobs exit codes.