pyslurmutils 1.5#

pyslurmutils provides SLURM utilities for scheduling jobs from Python.

pyslurmutils has been developed by the Software group of the European Synchrotron.

The main purpose of this library is to provide a concurrent.futures.Executor implementation for SLURM that can be used on any machine, not necessarily a SLURM client.

Important

SLURM REST API access requires authentication.

Before using any commands or Python APIs, you must provide a valid SLURM REST URL, user name, and access token.

See the Authentication Guide for instructions.

Getting Started#

Installation#

pip install pyslurmutils

Basic Usage#

from pyslurmutils.concurrent.futures import SlurmRestExecutor

with SlurmRestExecutor(
    url=url,                               # SLURM REST URL
    user_name=user_name,                   # SLURM user name
    token=token,                           # SLURM access token
    log_directory="/path/to/log",          # for log files (optional)
    data_directory="/path/to/data",        # TCP communication when not provided
    pre_script="module load ewoks",        # load environment (optional)
    parameters={"time_limit": "02:00:00"}  # SLURM job parameters (optional)
    python_cmd="python",                   # python command (python3 by default)
) as executor:
    future = executor.submit(sum, [1, 1])
    assert future.result() == 2

When submitting a task, a new SLURM job is scheduled by default.

Executor parameters#

The executor API provides these parameters to control the job scheduling:

  • max_workers (Default: None): limit the number of SLURM jobs that can be scheduled in parallel. None means unlimited.

  • max_tasks_per_worker (Default: 1): the SLURM job exits after a number of tasks. None means unlimited.

  • lazy_scheduling (Default: True): schedule SLURM jobs only when needed. Can only be disabled when max_workers is specified.

  • conservative_scheduling (Default: False): schedule the least amount of workers at the expense of tasks staying longer in the queue.

  • cleanup_job_artifacts (Default: False): cleanup job artifacts like logs.

  • initializer: execute when starting a job

  • initargs: parameters for initializer

  • initkwargs: parameters for initializer

Note

The terms “worker” and “job” are equivalent and used interchangeably in the documentation.