We have a MongoDB containing customers and transactions. In a number of queries we want to pull all of the transactions for a customer into our application to do some processing. As far as I'm currently aware we have two options with different caveats:
- Group by customer ID in an aggregation and
$pushthe transactions into the group object. This can fail if the customers transactions sums >16MB
- Return all transactions to the application and group by customer ID in memory before processing. This has the down-side of needing to pull a lot of data into memory.
Is there a pattern for solving this?
My idea would be to query without grouping so that our cursor would be a list of transactions, but somehow we retrieve chunks of transactions per customer ID. That way we would only ever have 1 customers transactions in memory and also not risk blowing up the document size in Mongo.