pyslurmutils |version|
======================

*pyslurmutils* provides SLURM utilities for scheduling jobs from Python.

*pyslurmutils* has been developed by the `Software group <https://www.esrf.fr/Instrumentation/software>`_ of the `European Synchrotron <https://www.esrf.fr/>`_.

The main purpose of this library is to provide a :code:`concurrent.futures.Executor` implementation for SLURM
that can be used on any machine, not necessarily a SLURM client.

.. important::

   **SLURM REST API access requires authentication.**

   Before using any commands or Python APIs, you must provide a valid SLURM REST URL, user name, and access token.

   See the :ref:`Authentication Guide <slurm-authentication>` for instructions.

Getting Started
---------------

Installation
~~~~~~~~~~~~

.. code-block:: bash

    pip install pyslurmutils

Basic Usage
~~~~~~~~~~~

.. code-block:: python

    from pyslurmutils.concurrent.futures import SlurmRestExecutor

    with SlurmRestExecutor(
        url=url,                               # SLURM REST URL
        user_name=user_name,                   # SLURM user name
        token=token,                           # SLURM access token
        log_directory="/path/to/log",          # for log files (optional)
        data_directory="/path/to/data",        # TCP communication when not provided
        pre_script="module load ewoks",        # load environment (optional)
        parameters={"time_limit": "02:00:00"}  # SLURM job parameters (optional)
        python_cmd="python",                   # python command (python3 by default)
    ) as executor:
        future = executor.submit(sum, [1, 1])
        assert future.result() == 2

When submitting a task, a new SLURM job is scheduled by default.

Executor parameters
+++++++++++++++++++

The executor API provides these parameters to control the job scheduling:

* :code:`max_workers` (Default: :code:`None`): limit the number of SLURM jobs that can be scheduled in parallel. :code:`None` means unlimited.
* :code:`max_tasks_per_worker` (Default: :code:`1`): the SLURM job exits after a number of tasks. :code:`None` means unlimited.
* :code:`lazy_scheduling` (Default: :code:`True`): schedule SLURM jobs only when needed. Can only be disabled when :code:`max_workers` is specified.
* :code:`conservative_scheduling` (Default: :code:`False`): schedule the least amount of workers at the expense of tasks staying longer in the queue.
* :code:`cleanup_job_artifacts` (Default: :code:`False`): cleanup job artifacts like logs.
* :code:`initializer`: execute when starting a job
* :code:`initargs`: parameters for :code:`initializer`
* :code:`initkwargs`: parameters for :code:`initializer`

.. note::

    The terms "worker" and "job" are equivalent and used interchangeably in the documentation.

.. toctree::
    :hidden:

    auth
    tutorials
    reference