I am quite new to Service Discovery and clustered systems. I started experimenting with Mesos and Marathon for the deployment of Docker containers, the Marathon REST API and UI seem to do a good job.
My problem is the actual discovery of deployed services. For testing purposes I deployed a Kafka Cluster scaled to 3 instances via Marathon, as so I did with a MongoDB test-cluster. The Mesos-DNS client gives me a record like kafka.marathon.mesos and mongo.marathon.mesos which implies the dynamically mapped port from the host to the container. The problem is, that my client explicitly needs information about the target port. Is there a general way to get those port information from the service automatically and dymanically? What about apps exposing multiple ports?
My thougts so far: - Doing a REST call to get a status about the deployed app and somehow extract the relevant data - Doing a DNS SRV lookup and somehow extract the relevant data - Having some kind of "master", statically bound to a port, with dynamic "clients".
I searched a lot for those informations but in the end most of the tutorials ended with a manual lookup which is not what I aim for.
You're spot on. I recently gave a presentation at XebiCon around this topic and plan to publish a blog post with details about the setup incl. GitHub repo. For starters you could have a look at a Python implementation for the HTTP API consumption part.
UPDATE: the blog post is now available here.