I ve installed dask. My main aim is clustering a large dataset, but before starting work on it, I want to make a few tests. However, whenever I want to run a dask code piece, it takes too much time and a memory error appears at the end. I tried their Spectral Clustering Example and the short code below.

Do you think what is the problem?

from dask.distributed import Client
from sklearn.externals.joblib import parallel_backend
from sklearn.datasets import make_blobs
from sklearn.cluster import DBSCAN

import datetime

X, y = make_blobs(n_samples = 150000, n_features = 2, centers = 3, cluster_std = 2.1)
client = Client()

now = datetime.datetime.now()
model = DBSCAN(eps = 0.5, min_samples = 30)
with parallel_backend('dask'):
print(datetime.datetime.now() - now)

1 Answers

MRocklin On Best Solutions

The Scikit-Learn algorithms are not designed to train over large datasets. They are designed to operate on data that fits in memory. This is described here: https://ml.dask.org/#parallelize-scikit-learn-directly

Projects like Dask ML do have other algorithms that look like Scikit-Learn, but are implemented differently that do support larger dataset sizes. If you're looking for clustering then you might be interested in this page to see what is currently supported: https://ml.dask.org/clustering.html