Dask worker resources for distributed workers

Question

When you define worker resources (http://distributed.readthedocs.io/en/latest/resources.html) in distributed multi-processing workers, is the resource pool defined for all processes?

For example, on the worker host I am running:

dask-worker --nprocs 8 --resources HOST=1

Now, if I submit task(s) that require resources={"HOST":1} does this guarantee that only one of the processes on that machine will execute this task at specific moment in time?

Answer 1

The resources keyword applies evenly to all processes. All workers will get a single HOST resource. In general for mature deployments on institutional clusters I recommend avoiding the --nprocs keyword and instead creating each dask-worker process individually; this is more explicit.

dask-worker scheduler-address:8786 --resources
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786
dask-worker scheduler-address:8786