Akka 2.7.0 memory leak?

Question

I'm using Akka actor system with version 2.7.0 and we are facing memory leak issue as below:

When the process gets completed, memory is not getting free and below are two potential causes: enter image description hereWe are using Akka Actors with sharding in a cluster. We are also using Cassandra for persistence. We basically create jobs and provide some tasks to the actors. To give you a rough idea, we create about 10K actors for a job over 20 minutes. We run 100 such jobs in parallel. We had following observations:

1. We were struggling with Cassandra connectivity after few mins, so we disabled remember-entities.
2. We have configured 16 vCPU and 32 GB RAM. After 1st run, the RAM stays around 4 to 6 GB. After the 2nd run, RAM stays around 6 to 8 GB. After the 3rd run of 100 jobs, memory utilization goes belong 13 GB and GC goes crazy. We have set Xmx to 16 GB.

https://github.com/akka/akka/issues/29977

Any possible resolution?

Answer 1

It's going to be hard for someone to effectively debug your application for you. A few things, I'd note however:

1. It's not a leak in Akka: it's a problem and/or leak in your application.
2. You mention you were struggling with Cassandra connectivity and you a large number of TagWriter objects. This seems odd to me, although I don't have system to run tests on so I can't be certain. But I'm very suspicious of this.
3. You don't talk about your sharding config. If you aren't passivating or stopping your sharded entities it's certainly expected to have memory growth.
4. Use Lightbend Telemetry. That might give you more information.

Answer 2

We found the issue. It was with cleanup.deleteAll. If we use deleteAllEvents, then memory gets released. Ushpinder has updated the ticket.