Kafka fs2 stream how to set a backpressure

100 views Asked by At

i have fs2 stream from kafka topic, is it possible to set, how much of data can be pulled? Be cause, currently i have a problem, if in the middle of a stream a problem occures, it end up with a memory spike. So basically to prevent this, i want to set, how much of data can a stream pull and process before pulling a new batch.

1

There are 1 answers

0
Daenyth On

Backpressure is automatic with fs2 because it's a pull-based system, not push-based. Streams request more data from their source when they need to use it.

You can control the message buffering during construction; see the docs here

withMaxPrefetchBatches looks like a relevant setting