Scaling AWS Kinesis with Spring binding

67 views Asked by At

I'm going to have three different microservices which one will produce a GPS data to AWS kinesis. Second and third one will consume it basically.

Since I'll have two different consumers I wanted to implement this via Kinesis streams. This's how I configure my producer:

spring.cloud.stream:
  kinesis:
    binder:
      autoCreateStream: false
      kplKclEnabled: true
      autoAddShards: false
  bindings:
    dataProducer-out-0:
      destination: ${KINESIS_DESTINATION:dataProducer}
      content-type: application/json
      producer:
        headerMode: none

But I can't get my head around this, if I don't provide a partitionExpression, it basically generates a random hash from payload to find a shard as I've read from documentation. There're a few topics here:

  1. Order is important for me based on two fields of my payload, how do I ensure this?
  2. If I want to scale these microservices and increase the instance count, how would that work? Would it cause any bottleneck?
  3. I see there's a delay while pushing messages, it it due to kpl -> RecordMaxBufferedTime. If so is there a way to handle this, I find using KPL is convenient due to the advantages it offers.
  4. When KPL pushes the payload I see a message complaining about it takes more than 500ms and I need to check my configuration, which configuration does it mean?

Thanks in advance!

0

There are 0 answers