I want to create a new column from an existing column by applying a function. If I try using dask or multiprocessing (as suggested here), I often get a "cannot pickle" error .

Say I have 16 cores on my machine. Correct me if I'm wrong, but shouldn't it should be possible to split my dataframe into 16 parts, do an apply on each using a separate core, and concatenate the 16 dataframes? Is there a function that does this? Is pickle an unavoidable bottleneck?

0 Answers