Basic example on how to use the CCC cluster¶

Basic example

When you’re running hundreds or thousands of jobs, automation is a necessity. This is where hopla can help you.

A simple example of how to use hopla on a CCC cluster. Please check the user guide for a more in depth presentation of all functionalities.

Imports¶

import hopla
from pprint import pprint

Executor Context¶

executor = hopla.Executor(
    cluster="ccc",
    folder="/tmp/hopla",
    queue="rome",
    image="/tmp/hopla/my-docker-img.tar",
    walltime=1,
    project_id="genXXX",
)

Submit Jobs¶

jobs = [
    executor.submit("sleep", k) for k in range(1, 11)
]
pprint(jobs)
print(jobs[0].delayed_submission)

[DelayedCCCJob(
  job_id=1,
  submission_id=None,
  _hub=n4h00001rs,
  image_name=/tmp/hopla/my-docker-img.tar,
),
 DelayedCCCJob(
  job_id=2,
  submission_id=None,
  _hub=n4h00001rs,
  image_name=/tmp/hopla/my-docker-img.tar,
),
 DelayedCCCJob(
  job_id=3,
  submission_id=None,
  _hub=n4h00001rs,
  image_name=/tmp/hopla/my-docker-img.tar,
),
 DelayedCCCJob(
  job_id=4,
  submission_id=None,
  _hub=n4h00001rs,
  image_name=/tmp/hopla/my-docker-img.tar,
),
 DelayedCCCJob(
  job_id=5,
  submission_id=None,
  _hub=n4h00001rs,
  image_name=/tmp/hopla/my-docker-img.tar,
),
 DelayedCCCJob(
  job_id=6,
  submission_id=None,
  _hub=n4h00001rs,
  image_name=/tmp/hopla/my-docker-img.tar,
),
 DelayedCCCJob(
  job_id=7,
  submission_id=None,
  _hub=n4h00001rs,
  image_name=/tmp/hopla/my-docker-img.tar,
),
 DelayedCCCJob(
  job_id=8,
  submission_id=None,
  _hub=n4h00001rs,
  image_name=/tmp/hopla/my-docker-img.tar,
),
 DelayedCCCJob(
  job_id=9,
  submission_id=None,
  _hub=n4h00001rs,
  image_name=/tmp/hopla/my-docker-img.tar,
),
 DelayedCCCJob(
  job_id=10,
  submission_id=None,
  _hub=n4h00001rs,
  image_name=/tmp/hopla/my-docker-img.tar,
)]
DelayedSubmission(
  command=sleep 1,
  execution_parameters=,
)

Generate a batch¶

jobs[0].generate_batch()
print(jobs[0].paths)
batch = jobs[0].paths.submission_file
with open(batch) as of:
    print(of.read())

/home/runner/work/hopla/hopla/examples/plot_ccc.py:52: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  jobs[0].generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

JobPaths(
  flux_dir=/tmp/hopla/logs/1_flux,
  job_id=1,
  log_folder=/tmp/hopla/logs,
  stderr=/tmp/hopla/logs/1_log.err,
  stdout=/tmp/hopla/logs/1_log.out,
  submission_file=/tmp/hopla/submissions/1_submission.sh,
  submission_folder=/tmp/hopla/submissions,
  task_file=/tmp/hopla/submissions/1_tasks.txt,
  worker_file=/tmp/hopla/submissions/worker.sh,
)
#!/bin/bash

# Parameters
#MSUB -q rome
#MSUB -Q long
#MSUB -m workflash,scratch,work
#MSUB -T 3600
#MSUB -n 1
#MSUB -c 1
#MSUB -E "--gres=gpu:0"
#MSUB -M 2000
#MSUB -r hopla
#MSUB -e /tmp/hopla/logs/1_log.err
#MSUB -o /tmp/hopla/logs/1_log.out
#MSUB -A genXXX
#MSUB -F #use the Flux plug

# Environment
echo $SLURM_JOB_ID
echo $HOSTNAME


# Command
pcocc-rs run n4h00001rs:/tmp/hopla/my-docker-img.tar  -- sleep 1
echo "HOPLASAY-DONE"

Start Jobs¶

We can’t execute the code on the CI since the CCC infrastructure is not available.

from hopla.config import Config

with Config(dryrun=True, delay_s=3):
    executor(max_jobs=2)

CCC_MSUB:   0%|          | 0/10 [00:00<?, ?it/s]
/home/runner/work/hopla/hopla/doc/../hopla/utils.py:354: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  self.generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

[command] ccc_msub /tmp/hopla/submissions/1_submission.sh

/home/runner/work/hopla/hopla/doc/../hopla/utils.py:354: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  self.generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

[command] ccc_msub /tmp/hopla/submissions/2_submission.sh

/home/runner/work/hopla/hopla/doc/../hopla/utils.py:354: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  self.generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

[command] ccc_msub /tmp/hopla/submissions/3_submission.sh

/home/runner/work/hopla/hopla/doc/../hopla/utils.py:354: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  self.generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

[command] ccc_msub /tmp/hopla/submissions/4_submission.sh

CCC_MSUB:  40%|████      | 4/10 [00:03<00:04,  1.33it/s]
/home/runner/work/hopla/hopla/doc/../hopla/utils.py:354: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  self.generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

[command] ccc_msub /tmp/hopla/submissions/5_submission.sh

/home/runner/work/hopla/hopla/doc/../hopla/utils.py:354: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  self.generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

[command] ccc_msub /tmp/hopla/submissions/6_submission.sh

CCC_MSUB:  60%|██████    | 6/10 [00:06<00:04,  1.07s/it]
/home/runner/work/hopla/hopla/doc/../hopla/utils.py:354: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  self.generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

[command] ccc_msub /tmp/hopla/submissions/7_submission.sh

/home/runner/work/hopla/hopla/doc/../hopla/utils.py:354: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  self.generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

[command] ccc_msub /tmp/hopla/submissions/8_submission.sh

CCC_MSUB:  80%|████████  | 8/10 [00:09<00:02,  1.23s/it]
/home/runner/work/hopla/hopla/doc/../hopla/utils.py:354: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  self.generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

[command] ccc_msub /tmp/hopla/submissions/9_submission.sh

/home/runner/work/hopla/hopla/doc/../hopla/utils.py:354: UserWarning: Can't import image: /tmp/hopla/my-docker-img.tar
  self.generate_batch()
   |Traceback (most recent call last):
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 125, in generate_batch
   |    self.import_image()
   |  File "/home/runner/work/hopla/hopla/doc/../hopla/ccc.py", line 175, in import_image
   |    stdout = subprocess.check_output(cmd)
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 466, in check_output
   |    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 548, in run
   |    with Popen(*popenargs, **kwargs) as process:
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1026, in __init__
   |    self._execute_child(args, executable, preexec_fn, close_fds,
   |  File "/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/subprocess.py", line 1955, in _execute_child
   |    raise child_exception_type(errno_num, err_msg, err_filename)
   |FileNotFoundError: [Errno 2] No such file or directory: 'pcocc-rs'

[command] ccc_msub /tmp/hopla/submissions/10_submission.sh

CCC_MSUB: 100%|██████████| 10/10 [00:12<00:00,  1.32s/it]
CCC_MSUB: 100%|██████████| 10/10 [00:15<00:00,  1.50s/it]

Total running time of the script: (0 minutes 15.309 seconds)

Estimated memory usage: 108 MB

Gallery generated by Sphinx-Gallery