blockrun.dask_runner

Dask cluster startup for local or SLURM execution.

The config’s first jobqueue key selects the cluster type: local -> LocalCluster, slurm -> SLURMCluster (scale submits sbatch). The client process must stay alive for the whole run; on exit the cluster (and its SLURM jobs) are torn down.

Config loading — bundled templates, layered overrides, key validation — lives in blockrun.dask_config.

Functions

start_dask(num_workers[, config_path, label, logger, ...]) → Iterator[dask.distributed.Client]

Start a Dask cluster from one or more YAML configs; yield a connected Client.

Module Contents

blockrun.dask_runner.start_dask(num_workers: int, config_path: str | Sequence[str] = 'dask-local', label: str = 'job', logger: logging.Logger | None = None, wait_timeout: int = 600, effective_config_path: str | None = None) → Iterator[dask.distributed.Client]

Start a Dask cluster from one or more YAML configs; yield a connected Client.

config_path is a bundled name (see bundled_configs()), a path, or a sequence of either, deep-merged left to right; unrecognised keys raise (see blockrun.dask_config). The default, dask-local, works from any directory.

The config’s first jobqueue key selects the cluster type:

• "local": a LocalCluster with num_workers single-threaded workers.

• "slurm": a SLURMCluster; scale(num_workers) submits sbatch jobs.

Tears the cluster (and its SLURM jobs) down on exit. effective_config_path (a directory or file) records the merged config, worth keeping with a run.