This module contains a ‘hopla’ function used to create a list of commands that will be executed in parallel using the local machine or cluster with TORQUE resource manager installed. For the moment, the input script must be a Python script.

hopla.converter.hopla(python_script, hopla_outputdir=None, hopla_cpus=1, hopla_logfile=None, hopla_verbose=1, hopla_cluster=False, hopla_cluster_logdir=None, hopla_cluster_queue=None, hopla_cluster_memory=1, hopla_cluster_walltime=24, hopla_cluster_python_cmd='python', hopla_iterative_kwargs=None, **kwargs)

Execute a python script/file in parallel.

Rules:

• This procedure enables local or cluster runs.
• This procedure returns a human readable log.
• In the command line, prefix one character kwargs with a ‘-‘, the other with a ‘–’.
• In order not to interfer with command line kwargs use ‘hopla’ prefix in function parameters.
• If the script contains a ‘__hopla__’ list of parameter names to keep trace on, all the specified parameters values are stored in the execution status.

Parameters:

python_script: str (mandatory)

a python script or file to be executed.

hopla_outputdir: str (optional, default None)

a folder where synthetic results are written.

hopla_cpus: int (optional, default 1)

the number of cpus to be used.

hopla_logfile: str (optional, default None)

location where the log messages are redirected: INFO and DEBUG.

hopla_verbose: int (optional, default 1)

0 - display no log in console, 1 - display information log in console, 2 - display debug log in console.

hopla_cluster: bool (optional, default False)

if True use a worker that submits the jobs to a cluster.

hopla_cluster_logdir: str (optional, default None)

an existing path where the cluster error and output files will be stored. This folder must be empty.

hopla_cluster_queue: str (optional, default None)

the name of the queue where the jobs will be submited.

hopla_cluster_memory: float (optional, default 1)

the memory allocated to each job submitted on a cluster (in GB).

hopla_cluster_walltime: int (optional, default 24)

the walltime used for each job submitted on the cluster (in hours).

hopla_cluster_python_cmd: str (optional, default ‘python’)

the path to the python binary.

hopla_iterative_kwargs: list of str (optional, default None)

the iterative script parameters.

kwargs: dict (optional)

the script parameters: iterative kwargs must contain a list of elements and must all have the same length, non-iterative kwargs will be replicated.

Returns:

execution_status: dict

a dictionary that contains all the executed command return codes.

exitcodes: dict

a dictionary with a summary of the executed jobs exit codes.