Basic example on how to use the PBS cluster

Basic example

When you’re running hundreds or thousands of jobs, automation is a necessity. This is where hopla can help you.

A simple example of how to use hopla on a PBS cluster. Please check the user guide for a more in depth presentation of all functionalities.

Imports

import hopla
from pprint import pprint

Executor Context

executor = hopla.Executor(
    cluster="pbs",
    folder="/tmp/hopla",
    queue="Nspin_short",
    image="/tmp/hopla/my-apptainer-img.simg",
    walltime=1
)

Submit Jobs

jobs = [
    executor.submit("sleep", k) for k in range(1, 11)
]
pprint(jobs)
print(jobs[0].delayed_submission)

[DelayedPbsJob(
  job_id=1,
  submission_id=None,
),
 DelayedPbsJob(
  job_id=2,
  submission_id=None,
),
 DelayedPbsJob(
  job_id=3,
  submission_id=None,
),
 DelayedPbsJob(
  job_id=4,
  submission_id=None,
),
 DelayedPbsJob(
  job_id=5,
  submission_id=None,
),
 DelayedPbsJob(
  job_id=6,
  submission_id=None,
),
 DelayedPbsJob(
  job_id=7,
  submission_id=None,
),
 DelayedPbsJob(
  job_id=8,
  submission_id=None,
),
 DelayedPbsJob(
  job_id=9,
  submission_id=None,
),
 DelayedPbsJob(
  job_id=10,
  submission_id=None,
)]
DelayedSubmission(
  command=sleep 1,
  execution_parameters=,
)

Generate a batch

jobs[0].generate_batch()
print(jobs[0].paths)
batch = jobs[0].paths.submission_file
with open(batch) as of:
    print(of.read())

JobPaths(
  flux_dir=/tmp/hopla/logs/1_flux,
  job_id=1,
  log_folder=/tmp/hopla/logs,
  stderr=/tmp/hopla/logs/1_log.err,
  stdout=/tmp/hopla/logs/1_log.out,
  submission_file=/tmp/hopla/submissions/1_submission.sh,
  submission_folder=/tmp/hopla/submissions,
  task_file=/tmp/hopla/submissions/1_tasks.txt,
  worker_file=/tmp/hopla/submissions/worker.sh,
)
#!/bin/bash

# Parameters
#PBS -q Nspin_short
#PBS -l mem=2gb,ncpus=1,ngpus=0,walltime=1:00:00
#PBS -N hopla
#PBS -e /tmp/hopla/logs/1_log.err
#PBS -o /tmp/hopla/logs/1_log.out

# Environment
echo $PBS_JOBID
echo $HOSTNAME

# Command
apptainer run  /tmp/hopla/my-apptainer-img.simg sleep 1
echo "HOPLASAY-DONE"

Start Jobs

We can’t execute the code on the CI since the PBS infrastructure is not available.

from hopla.config import Config

with Config(dryrun=True, delay_s=3):
    executor(max_jobs=2)

QSUB:   0%|          | 0/10 [00:00<?, ?it/s][command] qsub /tmp/hopla/submissions/1_submission.sh
[command] qsub /tmp/hopla/submissions/2_submission.sh
[command] qsub /tmp/hopla/submissions/3_submission.sh
[command] qsub /tmp/hopla/submissions/4_submission.sh

QSUB:  40%|████      | 4/10 [00:03<00:04,  1.33it/s][command] qsub /tmp/hopla/submissions/5_submission.sh
[command] qsub /tmp/hopla/submissions/6_submission.sh

QSUB:  60%|██████    | 6/10 [00:06<00:04,  1.06s/it][command] qsub /tmp/hopla/submissions/7_submission.sh
[command] qsub /tmp/hopla/submissions/8_submission.sh

QSUB:  80%|████████  | 8/10 [00:09<00:02,  1.23s/it][command] qsub /tmp/hopla/submissions/9_submission.sh
[command] qsub /tmp/hopla/submissions/10_submission.sh

QSUB: 100%|██████████| 10/10 [00:12<00:00,  1.32s/it]
QSUB: 100%|██████████| 10/10 [00:15<00:00,  1.50s/it]

Estimated memory usage: 108 MB