Distribute python for loops on cluster/grid

90 views Asked by At

Say I have a python script that looks like below. Since python is single threaded, what tool would I use to distribute the loops on different cores/computers?

series=pandas.Series()

# evaluate an model 
def evaluate_model(X, O):
    ...

def evaluate_models(dataset, p_values, d_values, q_values):
    dataset = dataset.astype('float32')
    best_score, best_cfg = float("inf"), None
    for p in p_values:
        for d in d_values:
            for q in q_values:
                order = (p,d,q)
                try:
                    mse = evaluate_model(dataset, order)
                    if mse < best_score:
                        best_score, best_cfg = mse, order
                    print('A%s MSE=%.3f' % (order, mse))
                except:
                    continue
    print('Best model%s MSE=%.3f' % (best_cfg, best_score))


# evaluate parameters
p_values = [0, 1, 2, 4, 6, 8, 10]
d_values = range(0, 3)
q_values = range(0, 3)

evaluate_models(series.values, p_values, d_values, q_values)

It would be nice if I could say:

run_on_grid(evaluate_models(series.values, p_values, d_values, q_values))

It would be nicer if "run_on_grid" could be configured to point to a local GPU.

0

There are 0 answers