How do I set Dask autoscaling using Iguazio?

Question

I need to create a new Dask cluster in Iguazio. I want to take advantage of Dask's autoscaling features that are described here: https://docs.dask.org/en/stable/how-to/adaptive.html

Does Iguazio support Dask cluster autoscaling and, if so, how do I enable that?

Answer 1

In the Iguazio, when you create a Dask cluster, you don't need to worry about lower-level dask_kubernetes related stuff. You just need to specify the min and max number of workers like below

# create an mlrun function which will init the dask cluster
dask_cluster_name = "dask-cluster"
dask_cluster = mlrun.new_function(dask_cluster_name, kind='dask', image='mlrun/ml-models')
dask_cluster.apply(mlrun.mount_v3io())

# set range for # of replicas with replicas and max_replicas
dask_cluster.spec.min_replicas = 1
dask_cluster.spec.max_replicas = 100

Depending on your workload, the cluster will scale up and down between the min and max number of workers. We bake in the adaptive deployments of the Dask cluster so it results in both faster analyses that give users much more power, but with much less pressure on computational resources.