Is there a way to apply backpressure to the whole pipeline of fluxes?

20 views Asked by At

I am using project reactor flux to read data from DB and then publish to a messaging system.

The flux consists of many steps, i.e.

// we have a collection of things initially in the sink

mainFlux = sink.asFlux()
            .filter(this::checkForEmptyDataEnvelopeCollection)
// do other transformations/checks here
            .flatMap(Flux::fromIterable)   // parallelize
            .map(...) // some prep work
            .map(mainPublisher::publish)   // publishing happens here
            .retryWhen( ... )  // publish to a different/error destination
            .subscribe(... )  // report completion here

The backpressure signal comes from the publisher (or from retry/error publisher).

What would be the best way to control backpressure for the whole pipeline?

The publishing is async, it returns CompletableFuture and I want to apply it if I have a certain number of futures in flight and I want to apply it to the whole pipeline - I want to stop getting things from the initial sink.

Do I have to have some kind of global state outside of flux?

0

There are 0 answers