We are building a service based on Kinesis / DynamoDB streams and one question we have (which we cannot find in the official documentation) is whether we can use the same lease table (DynamoDB) to store checkpoint information of different KCL applications that consume the same stream.
- Is it good practice?
- Can it generate some kind of inconsistency in behavior?
- Do you recommend using a separate lease table per KCL application?
Thank you very much.
DynamoDB lease table stores seq nums per shard for your application and can be used to restart the work in case your application restarts. It constantly updates the seq num for each shard once the next records processed.
That being said, if you for some reason decide to use the same lease table from different applications, then you will loose the possibility to gracefully recover the position in stream in case of failure and can get a data loss. In general, in this case the lease table will be useless as it will reflect the state of the most recently processed records, not even the state of a single application. The lease table state will be inconsistent in this case and messed.
To sum up - it's generally necessary to use separate lease table for different applications.