I am trying to put a PCollection<KV<String,POJO>> , I need to write this Pojo to GCS bucket in the avro format, where the string(key) for the map is the GCS folder path that I need to write into . How do I achieve this for streaming fashiom
I tried using AvroIO library but I am struggling to get the gcs folder path dynamically for every element of PCollection.
PCollection<KV<String, Pojo>> output; // this has the required data
output.apply(AvroIO.write(pojo.class).to(GCS Path)
.withSchema(SCHEMA).withSuffix(".avro"));
How do I retrieve the GCS path dynamically for every Pojo.