We have a MongoDB containing customers and transactions. In a number of queries we want to pull all of the transactions for a customer into our application to do some processing. As far as I'm currently aware we have two options with different caveats:

  1. Group by customer ID in an aggregation and $push the transactions into the group object. This can fail if the customers transactions sums >16MB
  2. Return all transactions to the application and group by customer ID in memory before processing. This has the down-side of needing to pull a lot of data into memory.

Is there a pattern for solving this?

My idea would be to query without grouping so that our cursor would be a list of transactions, but somehow we retrieve chunks of transactions per customer ID. That way we would only ever have 1 customers transactions in memory and also not risk blowing up the document size in Mongo.

0 Answers