kafka-avro-producer throws ClassCastException when serializing timestamp-millis values

78 views Asked by At

I'm using the "CLI Tools Shipped With Confluent Platform" and I'm trying to send a message to Kafka using the kafka-avro-producer like:

./kafka-avro-console-producer --bootstrap-server localhost:9092 --topic "my-topic" --property schema.registry.url=http://localhost:8081 --property value.schema.id=1234 < ./my-message.json

where the schema includes a timestamp field declared like:

{
    ..., # other fields
    {
        "name": "my_field_name",
        "type": {
            "type": "long",
            "logicalType": "timestamp-millis",
            "minimum": 0
        },
        "doc": "...."
    }
}

and the message (my-message.json) I'm sending uses this field as:

{
    ..., # other fields
    "my_field_name":1703162438282
}

For some reason the above command throws this error:

org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
        at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:174)
        at io.confluent.kafka.formatter.AvroMessageReader$AvroMessageSerializer.serialize(AvroMessageReader.java:167)
        at io.confluent.kafka.formatter.SchemaMessageReader.readMessage(SchemaMessageReader.java:406)
        at kafka.tools.ConsoleProducer$$anon$1$$anon$2.hasNext(ConsoleProducer.scala:67)
        at kafka.tools.ConsoleProducer$.loopReader(ConsoleProducer.scala:90)
        at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:99)
        at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: java.lang.ClassCastException: value 2023-12-21T12:40:38.282Z (a java.time.Instant) cannot be cast to expected type long at MyTopic.my_field_name
        at org.apache.avro.path.TracingClassCastException.summarize(TracingClassCastException.java:79)
        at org.apache.avro.path.TracingClassCastException.summarize(TracingClassCastException.java:30)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:84)
        at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.writeDatum(AbstractKafkaAvroSerializer.java:192)
        at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:162)
        ... 6 more
Caused by: java.lang.ClassCastException: java.time.Instant cannot be cast to java.lang.Number
        at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:174)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:93)
        at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:245)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:234)
        at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:145)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:95)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:82)
        ... 8 more

I've also tried using the string representation of the date which of course can't be serialized.

I've read online that this issue might be solved by setting this field as nullable in the schema registry, but I'm constrained by my client to have this field as not null. What should I do?

Edit: I've also seen this issue on GitHub which looks similar but it's about the Kafka Connect platform which I'm not using. Idk if that can be of any help.

0

There are 0 answers