This module proposes a local worker and a distant TORQUE worker. The two proposed workers are able to follow a ‘__hopla__’ list of parameter names to keep trace on. All specified parameters values are stored in the execution status.

hopla.workers.qsub_worker(tasks, returncodes, logdir, queue, memory=1, walltime=24, python_cmd='python', sleep=2)

A cluster worker function for a script.

Use the TORQUE resource manager provides control over batch jobs and distributed computing resources. It is an advanced open-source product based on the original PBS project.

Use a double script strategy in order to manage the ‘__hopla__’ list of parameter names to keep trace on: a ‘.pbs’ script calling another ‘.py’ script that print the ‘__hopla__’ parameters. All the specified parameters values are stored in the return code.

Parameters:

tasks, returncodes: multiprocessing.Queue

the input (commands) and output (results) queues.

logdir: str

a path where the qsub error and output files will be stored.

queue: str

the name of the queue where the jobs will be submited.

memory: float (optional, default 1)

the memory allocated to each qsub (in GB).

walltime: int (optional, default 24)

the walltime used for each job submitted on the cluster (in hours).

python_cmd: str (optional, default ‘python’)

the path to the python binary.

sleep: float (optional, default 2)

time rate to check the termination of the submited jobs.

hopla.workers.worker(tasks, returncodes)

The worker function for a script.

If the script contains a ‘__hopla__’ list of parameter names to keep trace on, all the specified parameters values are stored in the return code.

Parameters:

tasks, returncodes: multiprocessing.Queue

the input (commands) and output (results) queues.