Kafka Streams Reduce is returning a "Register operation failed because schema is too large; error code: 42209"

37 views Asked by At

I'm trying to group some messages' list by using .reduce to append the 2nd record to the 1st. However, I am getting a error as below.

Register operation failed because schema is too large; error code: 42209

The value serdes is using a reflection based avro generated schema based off a Java class.

        .groupByKey()
        .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMillis(100L)))
        .reduce((firstRecord, secondRecord) -> {
            firstRequest.getList().addAll(secondRequest.getList());
            return firstRecord;
        })
        .suppress(Suppressed.untilWindowCloses(new StrictBufferConfigImpl()))
        .toStream()

From my understanding, reduce isn't supposed to modify a schema, so I'm wondering how does this error happen?

It works without the reduce but obviously that's not useful since I need this reduce operation for my business usecase. Also, I checked my broker's settings and the max message in bytes is set to 10mb, which is plenty for this record.

0

There are 0 answers