I'm going to have three different microservices which one will produce a GPS data to AWS kinesis. Second and third one will consume it basically.
Since I'll have two different consumers I wanted to implement this via Kinesis streams. This's how I configure my producer:
spring.cloud.stream:
kinesis:
binder:
autoCreateStream: false
kplKclEnabled: true
autoAddShards: false
bindings:
dataProducer-out-0:
destination: ${KINESIS_DESTINATION:dataProducer}
content-type: application/json
producer:
headerMode: none
But I can't get my head around this, if I don't provide a partitionExpression
, it basically generates a random hash from payload to find a shard as I've read from documentation. There're a few topics here:
- Order is important for me based on two fields of my payload, how do I ensure this?
- If I want to scale these microservices and increase the instance count, how would that work? Would it cause any bottleneck?
- I see there's a delay while pushing messages, it it due to
kpl -> RecordMaxBufferedTime
. If so is there a way to handle this, I find using KPL is convenient due to the advantages it offers. - When KPL pushes the payload I see a message complaining about it takes more than
500ms
and I need to check my configuration, which configuration does it mean?
Thanks in advance!