Apache beam write PCollection<string,pojo> to GCS bucket

125 views Asked by At

I am trying to put a PCollection<KV<String,POJO>> , I need to write this Pojo to GCS bucket in the avro format, where the string(key) for the map is the GCS folder path that I need to write into . How do I achieve this for streaming fashiom

I tried using AvroIO library but I am struggling to get the gcs folder path dynamically for every element of PCollection.

PCollection<KV<String, Pojo>> output; // this has the required data
output.apply(AvroIO.write(pojo.class).to(GCS Path)
               .withSchema(SCHEMA).withSuffix(".avro"));

How do I retrieve the GCS path dynamically for every Pojo.

0

There are 0 answers