Apache Strom upgrade from 1.0.3 to 2.2.0 and not all workers are used

Question

I am upgrading from Apache Storm 1.0.3 to 2.2.0 and facing a peculiar issue where the topology tasks are not running on all the workers and is only running on the same worker as the Spout.

I am using shuffleGrouping and have configured 6 workers and a total of 51 executors among 2 bolts and 1 spout. When I start the topology all 6 the workers are started as expected but only the one worker with the spout is doing all the work.

Do I need any special configuration for 2.2.0, all my topologies are working fine with version 1.0.3

As a test I am also running the ExclamationTopology from Strom-starter but seeing the same issue of only one worker doing all the work.

Thanks in advance for any help.

Regards,

Answer 1

When upgrading to storm 2.0.0 I was baffled by the behavior you have observed as well. The reason lies within the shuffleGrouping which you are using. According to the storm performance guide:

6. Load Aware messaging

When load aware messaging is enabled (default), shuffle grouping takes additional factors into consideration for message routing. Impact of this on performance is dependent on the topology and its deployment footprint (i.e. distribution over process and machines). Consequently it is useful to assess the impact of setting topology.disable.loadaware.messaging to true or false for your specific case.

This leads to the spout delegating all tuples to the same worker it finds itself on. To fix it and to go back to the old behavior, in your topology, set topology.disable.loadaware.messaging to true, e.g. with

conf.put("topology.disable.loadaware.messaging", true);