We have a topology with topology.message.timeout.secs = 300 secs. Recently, we ran into an issue where the first bolt after the spout reached a capacity of ~2. The bolt started processing tuples very slowly (it started processing tuples 5 mins after the spout emitted them). After a few minutes, the spout emitted tuples but the before the bolt could start processing, the tuple timed out.
The strange part is the time difference between when the tuple failed vs when it was emitted is 10 mins.
The expectation was that the tuple should have failed after 5 mins (300 secs configured).
Any thoughts/information on why the timeout configured was not really considered would be very helpful. Is there any other configuration that can affect the tuple timeout?