Basically, I don't need output from DoFn, just want to update some mysql db for each record I am getting in DoFn. So how could I define DoFn having void data type ? Basically I don't want to emit anything form DoFn.
How could I define the DoFn in apache crunch having "void" data type?
279 views Asked by Vivek Rai At
1
it is not possible. Once Crunch generate the graph for the execution, it determines if there is or not an output to be written or data to be materialized (pretty similar to a action). Even to accomplish what you want, you will need to do a workaround like create a
DoFn<T,String>are code for write or materialize the PCollection result from your DoFn, even if you already know that there is not going to be any outputs.