Increasing assigned memory for a topology in Storm

Question

I have a 10 node cluster with each machine of 8 GB RAM and when I run my topology, the assigned memory is always proportional to the number of workers. And each worker is approximately taking a 1 GB of Memory. I want to allot 2 GB to each worker I tried to set in storm.yaml as worker.childopts: "-Xmx6g -Xms6g" since I am running three workers in each node. But the assigned memory decreased to below 1 GB.

How to tune my topology better?

I am getting the following error in one of my bolts

java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.kafka.common.memory.MemoryPo

Answer 1

I think your configuration is slightly wrong. The worker.childopts setting is passed to each worker JVM, so when you set -Xmx6g -Xms6g you are giving each of your three workers 6 gigs of memory (18 total for the node)

Answer 2

You can reference this link https://github.com/apache/storm/blob/master/conf/defaults.yaml

I config 2 params:

worker.heap.memory.mb: 768
supervisor.memory.capacity.mb: 4096.0

If you have 10 workers you have to config supervisor.memory.capacity.mb = (768 x 10) It's capacity of supervisor