Apache Beam Publish Kafka Message with KafkaIO and KafkaAvroSerialization for GenericRecord

33 views Asked by At

I am new to Beam and I don't know how can I publish custom message using KafkaIO.write. I have to use KafkaAvroSerializer.class as the value serializer.

public void publish(CustomObject customObj) {
    PipelineOptions options = PipelineOptionsFactory.create();

    Pipeline p = Pipeline.create(options);


  PCollection<KafkaRecord<Long, GenericRecord>> response = 
  p.apply(Create.of(customObj))
     .apply(KafkaIO.<Long, GenericRecord>write()
            .withBootstrapServers("localhost:9092")
            .withTopic("myTopic")
            .withKeySerializer(LongDeserializer.class)
            .withValueSerializer(KafkaAvroSerializer.class)

            .updateConsumerProperties(ImmutableMap.of("auto.offset.reset", (Object)"earliest"))

           
    );
    p.run().waitUntilFinish();
}
0

There are 0 answers