heroku R14 errors with java Play 1 app running on OpenJDK 1.8

Question

I recently updated a Play v1 app to use OpenJDK v1.8 on Heroku and am finding that after typical load I start getting R14 errors relatively quickly - the physical memory has exceeded the 512MB limit and is now swapping impacting performance. I need to restart the applications frequently to stop the R14 errors. The application is a very typical web application and I would expect it to run comfortably within the memory constraints.

Here is a screenshot from NewRelic showing the physical memory being exceeded. I don't really have 59 JVMs, just the result of numerous restarts.

I don't quite understand why the "used heap" appears to impact the "physical memory" when it hasn't come close to the "committed heap" and why the "physical memory" seems to more closely follow the "used heap" rather than the "committed heap"

I've used the Eclipse Memory Analyzer to analyse some heap dumps and the Leak Suspects report mentions play.Play and play.mvc.Router as suspects though I'm not sure if that's expected and/or if they are directly related to the physical memory being exceeded.

See the generated by MAT for more details.

• Leak Suspects
• Top Components

Any guidance on how to resolve this would be great. I'm developing on OS X with Oracle Java 1.8 and have not yet been able to replicate the exact Heroku dyno environment locally (e.g. Ubuntu, OpenJDK 1.8) to attempt to reproduce the issue.

UPDATE 11/12/2014:

Here is the response from Heroku Support:

---

Play, and in turn Netty, allocate direct memory ByteBuffer objects for IO. Because they use direct memory, they will not be reported by the JVM (Netty 4.x now uses ByteBuf objects, which use heap memory).

The 65MB of anonymous maps that your app is using is not terribly uncommon (I’ve seen some use 100+mb). Some solutions include:

1. Limiting the concurrent of your application (possibly by setting play.pool)
2. Increase your dyno size to a 2X dyno.
3. Decrease your Xmx setting to 256m. This will give the JVM more room for non-heap allocation.

Please let us know if these solutions do not work for you, or if you continue to experience problems after adopting them.

If you want to reproduce the Heroku environment locally, I recommend installing Docker, and using this Docker image with your application. Let us know if you have any trouble with this.

---