I use streamparse for a while now but i'm stuck on one subject.
We use storm-0.10.0 and streamparse==2.1.4. We let all the default value (no auto_anchor = False or something like that ) .
We have no ack or fail method implemented in spout and we don't call any ack or fail method in bolts .
On bolt side, process latency and execute latency looks correct . Process latency are >> execute latency.
My concern is about the spout Complete Latency that we can find in the storm UI . It's always at 0.00 . I read that complete Latency is the total time spent of a message in the topology .
So it's a very good KPI to follow as it can help to detect a lake of workers, tasks or even nodes.
In one bolt , i did a self.log(tup) and i do have a tuple.id with this kind of output :
name:engine-bolt Tuple(id=u'2213630928741732943',
component=u'request-spout-1',
stream=u'default',
task=24,
values= etc
...
So how can i do to get some values about the "Complete Latency" for spouts ?
If you need anything else , please ask :)