I'm working with some code that I plan on running on a server in the near future. Right now it works on my local machine, but multiple people will be running the program at the same time. I'm worried that they will use more ram or vram than available. If I use dask will it wait for available resources before executing the function call?
#!/usr/bin/env python3 # -*- coding: utf-8 -*- from numba import njit import numpy as np from dask.distributed import Client, LocalCluster @njit def addingNumbers (big_array, big_array2, save_array): for i in range (big_array.shape): for j in range (big_array.shape): save_array[i][j] = big_array[i][j] * big_array2[i][j] return save_array if __name__ == "__main__": cluster = LocalCluster() client = Client(cluster) big_array = np.random.random_sample((100, 3000)) big_array2 = np.random.random_sample((100, 3000)) save_array = np.zeros(shape=(100, 3000)) x = client.submit(addingNumbers, big_array, big_array2, save_array) y = client.gather(x)
If multiple people were running the above code at the same time and the server was almost out of ram, would dask wait until ram was available to submit the function, or would it submit it and the server would get an out of memory error?
If dask doesn't wait till ram is available, how would you queue the function call? Thanks