I have an algorithm where in a flowfile I create a queue of flowfile uuids that I must process from top to bottom. uuids are written to this flowfile seperated by newlines, so that a uuid at each line represents the next flowfile to process.This is our notify signal to a wait processor.
This flow file is cached based on the id of the object I am processing. So for this object with a given id, I can only process one flow file at a time from the queue of uuids from top to bottom.
There are multiple such cached flow file based on my data object id. So effectively for any given data object with a certain id, only one flow file can be in progress. Different dataobjects have different ids allow parallelism in my flow. This is besides the point that I have but this sums up my algorithm.
So to the problem at hand. Once a flowfile goes to completion for a give data object for a certain id, I remove the uuid of this flow file from the top of the queue. Cache this flow file back (in the distributed cache server). Then get the next uuid to process and notify this.
Now I am tackling failure scenarios and one scenario that theoretically could happen is once a flow file for a given data object id finishes, it will come back to delete its uuid from the head of the queue. Once deleted, it will pick up the next uuid in the queue and notify it. So notify for the next uuid has happened now.
But meanwhile the flowfile with this uuid would have expired and no longer be in the wait processor.
This expired flowfile also comes to the queue and it searches for its uuid and deletes it from the queue. For expired/failure relationships from the wait processor, they can delete the uuids from anywhere in the queue, as expiry is going to happen for messages that have been waiting in the queue. At this point the job is done for expired messages.
But now we have this hanging notify signal in the wait processor.
Question is, can the notify processor with a delta value of zero, clear this fired signal for this data object id in the wait processor? The cleanup happens out of order, i.e notify has happened as already mentioned and I come back with my expired flowfile later on and only than can do this cleanup. Does this cleanup has any effect at this point in time i.e cancels the signals for the non existent flowfile as it has moved on to the expired relationship ?
Are these dangling notify signals heavy on the wait processor?