I want to run multiple Kafka brokers on Mesos as Marathon jobs. I'm thinking what to use as a broker ID. I know that some people are using IP based broker ID, but I read that this approach is not optimal for situations when migrating the brokers to different machines. Any insights?
Related to that, is there a way to do a rolling deploy of Kafka brokers in a way that Marathon waits for the old broker instances to replicate the data to the new ones before killing them.
I created a Docker image which can start on Marathon and also scale up and down. To get a unique broker id, I concatenate the last octet of the IP address and the port number which Marathon devises to the Docker container.
I understood that this shouldn't be a problem, because normally the clients connect via ZooKeeper, which itself has the nodes and their connection info, so you shouldn't need to take care of the details. Or I misunderstood the question...
Have a look at