Pmap inside a transducer in Clojure

Question

I would like to use pmap within a transducer, if I write my code using regular map, it works fine. However if I use pmap I get an arity exception. Is it possible to achieve this in clojure or am I doing something wrong? If it were to be impossible, could someone point to the documentation on why that happens? just out of curiosity. Pasted below is my code and the beginning of the arity exception.

(def get-in-map (comp
              (pmap read-csv)
              ))
clojure.lang.Compiler$CompilerException: clojure.lang.ArityException: Wrong number of args (1) passed to: core/pmap, compiling:

Answer 1

pmap doesn't support creating a transducer with it. If you think about it, it makes sense. The parallelisation you get from pmap vs map breaks the guarantees of a transducer. It complects a parallel context with the description of the process. As described here transducers separate out the consuming context from the code that describes the task to be accomplished.

The core implementation of a parallel context for transducers is as yet unimplemented: see this ticket. However, you could use core.async/pipeline or core.reducers/fold to create one yourself.

Answer 2

Whereas map is documented to "[return] a transducer when no collection is provided," (doc), that is not the case for pmap (doc).

You can see the implementation of the single-argument version of map here. As you see here, pmap does not have a unary version, so that's why you get the ArityException.

That begs the question: why is there no unary pmap? Because transducers can provided that parallelism in a more general way. When you use transducers, there is no need to implement parallelism in each operation (parallel version of map, etc.). Look at clojure.core.reducers/fold: it "reduces a colletion using a (potentially parallel) reduce-combine strategy." Example (from David Nolen):

(clojure.core.reducers/fold + 
                            ((map inc) +) 
                            (vec (range 1000000)))