pyslurmutils 1.0

pyslurmutils provides SLURM utilities for scheduling jobs from Python.

pyslurmutils has been developed by the Software group of the European Synchrotron.

The main purpose of this library is to provide a concurrent.futures.Executor implementation for SLURM that can be used on any machine, not necessarily a SLURM client.

pip install pyslurmutils

from pyslurmutils.concurrent.futures import SlurmRestExecutor

with SlurmRestExecutor(
    url=url,                               # SLURM REST URL
    user_name=user_name,                   # SLURM user name
    token=token,                           # SLURM access token
    log_directory="/path/to/log",          # for log files (optional)
    data_directory="/path/to/data",        # TCP communication when not provided
    pre_script="module load ewoks",        # load environment (optional)
    parameters={"time_limit": "02:00:00"}  # SLURM job parameters (optional)
    python_cmd="python",                   # python command (python3 by default)
) as executor:
    future = executor.submit(sum, [1, 1])
    assert future.result() == 2

When submitting a task, a new SLURM job is scheduled by default.

The executor API provides these parameters to control the job scheduling:

• max_workers (Default: None): limit the number of SLURM jobs that can be scheduled in parallel. None means unlimited.

• max_tasks_per_worker (Default: 1): the SLURM job exits after a number of tasks. None means unlimited.

• lazy_scheduling (Default: True): schedule SLURM jobs only when needed. Can only be disabled when max_workers is specified.

• conservative_scheduling (Default: False): schedule the least amount of workers at the expense of tasks staying longer in the queue.

• cleanup_job_artifacts (Default: True): cleanup job artifacts like logs.

• initializer: execute when starting a job

• initargs: parameters for initializer

• initkwargs: parameters for initializer

Note

The terms “worker” and “job” are equivalent and used interchangeably in the documentation.