I have developed a StormCrawler project that has multiple additional Bolts in that topology. My crawler should work 7 X 24 without any downtime. So I can not restart the crawler and change the topology configuration. I want to bypass (turn on or turn off) some bolts during runtime. What is the best way to disable and enable some Bolts in StormCrawler at runtime?
thanks
There is no way of doing it out of the box, so you'd have to implement the logic for tuning the bolts on / off in the bolts themselves.
If what you need is to refresh their configuration, you could implement a dynamic mechanism. For instance, store the config of the bolts in for instance an Elastic index and reload that config periodically.
We already have something a bit like this with the JSONURLFilterWrapper and the equivalent for the ParseFilter. We could have an abstract ES-backed dynamically configurable bolt. Feel free to open an issue on GitHub if you think this is of interest or even better, contribute a PR ;-)