How to balance kinesis shards across several record processor?

Question

I am currently writing the simple Kinesis Client Library (KCL) in Golang version. One of the features that I want it for my simple KCL is load balancing shards across multiple record processors and EC2 instances. For example, I have two record processors (which will run in the separate EC2 instance) and four Kinesis shards. The load balancing feature will allow each record processors to process two Kinesis shards.

I read that Java KCL implemented this but I can't find the implementation in the library. My question is how am I going to implement this feature in Golang? Thank you.

Answer 1

Before you start writing your own client...it looks like there are some people who have already done this:

• https://github.com/nieksand/gokinesis
• https://github.com/runtakun/kinesis-connector-go

Another option you have is the KCL MultiLangDaemon. You can install a small runner program that does all the balancing for you, and then you just listen to the messages the daemon sends you and commit them back.

https://github.com/awslabs/amazon-kinesis-client#amazon-kcl-support-for-other-languages

Answer 2

For people interested in the topic, VMWare wrote the Go implementation of the Java lib: https://github.com/vmware/vmware-go-kcl/

At the time of writing, though, it doesn't support lease stealing: https://github.com/vmware/vmware-go-kcl/issues/4

Answer 3

KCL already does load balancing for you.

Here's a basic description of how it works today (keep in mind this is just the basics and is subject to change as Amazon improves the logic):

• When a worker (which may process multiple shards) starts up, it checks a central DynamoDB database for which shards are owned by workers (creating that database if necessary). This is the "leasing" table.
  • A "lease" is a relationship between a worker and a shard
  • Workers will process records for shards it owns an unexpired lease for
  • Leases expire if the worker hasn't emitted a "heartbeat" for the lease before it expires (typically every few seconds) - this heartbeat essentially updates the DDB record
• It checks Kinesis stream for which shards are available, and updates the table if needed
• If any leases are expired, the worker will try to take ownership of the lease - at database level, use shardId as key and write it's workerId there.
• If a worker starts and all shards are already taken, it checks to see what the "balance" is - if it detects an imbalance (ie: "I own 0 shards and some other worker owns 10 shards"), it initiates a "steal shard" protocol - the old worker stops processing for that shard, and the new worker starts

You're of course free to examine the source code for KCL on github: https://github.com/awslabs/amazon-kinesis-client - hopefully this explanation gives you more context for how to understand KCL and adapt it to your needs.