How to generate several Enumerators out of a single Enumerator (partition, split, ..)

Question

Is it possible to create several Enumerators out of a single Enumerator ? What I'm looking for is the equivalent of List.partition that returns a (List[A], List[A]) , such as

List().partition(_.age >= 18) 

So ideally, I'd like to apply a transformation on an Enumerator that allows me to partition the data so that I get a (Enumerator[T], Enumerator[T]) pair.

Is that easily doable using Play's Iteratee API ?

The closest thing I found is the Enumeratee.grouped method, which allows to group inputs, but as far as I understand, if all I want is 2 partitions (groups), I'll have to consume the entire Enumeratorto get a result. But I'd like the resulting Enumerators to be fed asynchronously from the input.

Answer 1

Unlike List, Enumerator doesn't represent a collection of data, but rather a data source that an Iteratee can be connected to. Data is streamed to the Iteratee as the Iteratee consumes the input. Because of this, it isn't well defined how the two Enumerator instances returned by this Enumerator.partition would behave. If the Iteratee consuming the first Enumerator is ready for more input, but the second isn't, what happens? Does the first one just have to wait? Does data get buffered up for the second one? What happens if one of the two Iteratee instances says it doesn't need any more input, or dies with an error? Do we kill the other one? Do we let it keep going, throwing away the data originally meant for the dead Iteratee?

Because of this, there can't really be a single canonical partition method, like there is for List. You could certainly write one, that behaves the way you want it to, but in actual practice it might be easier to have a single Iteratee do all the processing. If you are looking for partition because you want to run the two streams in parallel, you probably want to use Akka. If you just want to have the pipelined parallelism that the Iteratee API provides, you could have an Enumeratee[Whatever, Either[Whatever]] that would allow later Enumeratee instances to only convert the part of the stream that they need to.

If you are willing to indulge in the black magic that is monad transformers, then I think there might be a way to use something like Haskell's EitherT. But don't do that.