Note

This page is a reference documentation. It only explains the class signature, and not how to use it. Please refer to the user guide for the big picture.

hopla.Executor¶

class hopla.Executor(cluster, folder, queue, image, name='hopla', memory=2, walltime=72, n_cpus=1, n_gpus=0, n_multi_cpus=1, modules=None, project_id=None, backend='flux')[source]¶

Bases: object

Base job executor.

Parameters:

cluster: str: the type of cluster: ‘slurm’, ‘ccc’, ‘pbs’.
folder: Path/str: folder for storing job submission/output and logs.
queue: str: the name of the queue where the jobs will be submitted.
image: str: path to a docker ‘.tar’ image or apptainer ‘.simg’ image or name of an existing image.
name: str, default ‘hopla’: the name of the submitted jobs.
memory: float , default 2: the memory allocated to each job (in GB).
walltime: int default 72: the walltime used for each job (in hours).
n_cpus: int, default 1: the number of cores allocated for each job.
n_gpus: int, default 0: the number of GPUs allocated for each job.
n_multi_cpus: int, default 1: the number of cores reserved for each multi-tasks job.
modules: list of str, default None: the environment modules to be loaded.
project_id: str, default None: the project ID where you have computing hours.
backend: str, default ‘flux’: the multi-taks backend to use: ‘flux’, ‘joblib or ‘oneshot’. This option is only used with CCC cluster type.

Raises:

ValueError: If the cluster type is not supported.

Examples

>>> import hopla
>>> executor = hopla.Executor(
...     cluster="slurm",
...     folder="/tmp/hopla",
...     queue="Nspin_long",
...     image="/tmp/hopla/my-apptainer-img.simg",
... )
>>> jobs = [executor.submit("sleep", k) for k in range(1, 11)]
>>> executor(max_jobs=2) 
>>> print(executor.report) 

__init__(cluster, folder, queue, image, name='hopla', memory=2, walltime=72, n_cpus=1, n_gpus=0, n_multi_cpus=1, modules=None, project_id=None, backend='flux')[source]¶

property n_done_jobs¶: Get the number of finished jobs.

property n_jobs¶: Get the number of stacked jobs.

property n_running_jobs¶: Get the number of running jobs.

property n_waiting_jobs¶: Get the number of waiting jobs.

property report¶: Generate a general report for all jobs.

property status¶: Display current status.

submit(script, *args, execution_parameters=None, **kwargs)[source]¶

Create a delayed job.

Parameters:

script: Path/str or list of DelayedSubmission: script(s) to execute.
*args: any positional argument of the script.
execution_parameters: str: parameters passed to the container during execution.
**kwargs: any named argument of the script.

Returns:

job: DelayedJob: a job instance.

Raises:

RuntimeError: If the job class is not DelayedCCCJob for multi-tasks submission.

Examples using `hopla.Executor`¶

Basic example on how to use the CCC cluster

Basic example on how to use the CCC cluster using multi-tasks

Basic example on how to use the PBS cluster

Basic example on how to use the SLURM cluster

hopla.Executor¶

Examples using hopla.Executor¶

Examples using `hopla.Executor`¶