AWS EMR Hadoop Mapreduce physical memory limit error

2.3k views Asked by At

I keep getting this error when running some of my steps: Container [pid=5784,containerID=container_1482150314878_0019_01_000015] is running beyond physical memory limits. Current usage: 5.6 GB of 5.5 GB physical memory used; 10.2 GB of 27.5 GB virtual memory used. Killing container.

I searched over the web and people say to increase the memory limits. This error is after I already increased to the maximum allowed on the instance I'm using c4.xlarge. Can I get some assistance about this error and how to solve this?

Also, I don't understand why mapreduce will throw this error and won't just swap or even work slower but just continue to work ...

NOTE: This error started happening after I changed to a custom output compression so it should be related to that.

Thanks!

0

There are 0 answers