Running Apache HOP generated jar file over remote Apache Flink (org.apache.beam.runners.core.construction.SerializablePipelineOptions)

Question

My question is about running Apache HOP generated jar file over remote Apache Flink.

We are facing a problem with Java Serial version UID. We guess that configuration of Apache Beam serialized file’s serial version uid doesn’t match with the one at flink cluster.

How could we overcome this problem, we tried using same JDK implementations over the flink server and local machine (which generates the jar).

Recompiling the Beam Project, with specific serialversionuid’s , might be a solution, but we dont want to use this approach, because it might cause side-effects.

Could you please support us with an alternative solution

Thanks.

enter image description here

enter image description here

enter image description here

Answer 1

We have received multiple issues around running things on Spark and Flink. A step-by-step guide was created to run the samples.

In these samples we use Flink 1.13.5 (same as your version)

If the samples work on your cluster but your current pipeline does not it is also possible that one of the specific transforms you are using are causing the issue, if this is the case can you provide us with a sample pipeline to reproduce the issue so we can have a closer look.

Feel free to reach out to us using our mailinglists.

Cheers,

Hans