GenerateTableFetch drops flowfile on no data for query

Question

I am using using GenerateTableFetch with incoming flow files containing table_name to import data. And I have put Additional WHERE clause to handle incremental updates using updated_at column of the table manually.

Issue is when there is no data to import for the constructed query, it simply drops the flow file. It should have a separate relationship other than success and failure for this case so that I can handle on my own if something has to be done for such cases.

Currently, at the start of the workflow, I am using Wait on number of tables I have put into flow and tracking when they are done to increment the done count using Notify. But for the tables which get's dropped in GenerateTableFetch, it just keeps on waiting.

Any way to handle this?

Answer 1

This would indeed be a good improvement to add to GenerateTableFetch, perhaps an optional property to "Output Empty FlowFile on Zero Results" or something. I have written NIFI-5604 to cover this improvement. Combining this with NIFI-5601, you could do routing downstream (if fragment.count = 0 for example).