I'm trying to group some messages' list by using .reduce to append the 2nd record to the 1st. However, I am getting a error as below.
Register operation failed because schema is too large; error code: 42209
The value serdes is using a reflection based avro generated schema based off a Java class.
.groupByKey()
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMillis(100L)))
.reduce((firstRecord, secondRecord) -> {
firstRequest.getList().addAll(secondRequest.getList());
return firstRecord;
})
.suppress(Suppressed.untilWindowCloses(new StrictBufferConfigImpl()))
.toStream()
From my understanding, reduce isn't supposed to modify a schema, so I'm wondering how does this error happen?
It works without the reduce but obviously that's not useful since I need this reduce operation for my business usecase. Also, I checked my broker's settings and the max message in bytes is set to 10mb, which is plenty for this record.