storm v1.2.2 I have an environment with 6 node cluster each 16 cores. I have deployed a sample topology which reads messages from a kafka topic which has 12 partitions.
Configurations
Max spout pending is 15. Topology message timeout is 300 seconds Topology Acker executor is 2(equal to number of workers). Worker1 -> 15 executors for Bolt-A (task is equal to number of executors) Worker2 -> 15 executors for Bolt-A (task is equal to number of executors) bolts(A to B to C) each has 30 executors across 2 workers
I tried to publish 20 messages to that kafka topic, since max spout pending is 15 so only 15 messages should have fetched and emitted from spout but what i figured out was only 7 messages were emitted to Bolt-A. After these 7 messages were emitted to Bolt-B then Bolt-A received remaining messages from spout.
So ideally bolt-A should get 15 messages from spout and once these 15 are acknowledged then remaining 5 messages should have picked up. But i saw only 7 messages were picked up and i could see 8 executors are sitting idle and doing nothing whether i am missing something in storm configuration? I tried with localOrSuffled and Suffled grouping but again few executors are idle always. Any suggestions to equally distribute work to all available executors?