Flink stream going to two sinks based on conditions

Question

Trying to see the possibility of stream going to two sinks based on conditions.

Requirement is stream have events, all events after transformation need to go to one sink ( assume one kafka topic)

And only error events needs to go to another sink ( assume another kafka topic).

did not see use-case of once transformation is done , additional logic putting in sink. Looking if something similar done

Answer 1

The best way to do this is with side outputs.

private static final OutputTag<String> errors = new OutputTag<>("errors") {};

...

// in your main() method
SingleOutputStreamOperator<T> result = events.process(new ProcessFunction());

result.addSink(sink).name("normal output");
result.getSideOutput(errors).addSink(errorSink).name("error output");

...

// in the process function

if (somethingGoesWrong) {
    ctx.output(errors, "error message");
}

While there are other ways to split a stream with Flink, side outputs are very flexible (e.g., the side outputs can have different types) and perform well.