pyslurmutils 0.3#

pyslurmutils provides SLURM utilities for scheduling jobs from Python.

pyslurmutils has been developed by the Software group of the European Synchrotron.

The main purpose of this library is to provide a concurrent.futures.Executor implementation for SLURM that can be used on any machine, not necessarily a SLURM client.

pip install pyslurmutils

from pyslurmutils.concurrent.futures import SlurmRestExecutor

with SlurmRestExecutor(
    url=url,                               # SLURM REST URL
    user_name=user_name,                   # SLURM user name
    token=token,                           # SLURM access token
    log_directory="/path/to/log",          # for log files (optional)
    data_directory="/path/to/data",        # TCP communication when not provided
    pre_script="module load ewoks",        # load environment (optional)
    parameters={"time_limit": "02:00:00"}  # SLURM job parameters (optional)
    python_cmd="python",                   # python command (python3 by default)
) as executor:
    future = executor.submit(sum, [1, 1])
    assert future.result() == 2

When submitting a task, a new SLURM job is scheduled by default.

The executor API provides these parameters to control the job scheduling:

• max_workers (Default: None): limit the number of SLURM jobs that can be scheduled in parallel. None means unlimited.

• max_tasks_per_worker (Default: 1): the SLURM job exits after a number of tasks. None means unlimited.

• lazy_scheduling (Default: True): schedule SLURM jobs only when needed. Can only be disabled when max_workers is specified.

• conservative_scheduling (Default: False): schedule the least amount of workers at the expense of tasks staying longer in the queue.

• cleanup_job_artifacts (Default: True): cleanup job artifacts like logs.

• initializer: execute when starting a job

• initargs: parameters for initializer

• initkwargs: parameters for initializer