I have a mesos cluster with marathon and everything running in docker. I want to deploy production mysql and mongodb servers and am trying to figure out the best way to do service discovery.
Option 1 is to use the existing setup which proxies everything through an haproxy server sitting on one of the masters. My issue with this is that every query must make two hops, from app machine to master machine to db machine, even if the query originated on the machine the db is on. Is this actually significant or am I being pedantic?
Option 2 is to use mesos dns which I have set up but never used. This would require running the mysql docker image in host networking mode or adding 3306 to the ports a mesos slave can offer then immediately claiming 3306. I would then be able to use mysql.marathon.mesos as the host of any connection.
Option 3 would be to deploy the same haproxy server on every machine and use an app's local haproxy to send the queries to the right place. This seems a little bit better than option 1 because the first hop is always on the same machine but will require significantly more work to set up.
Has anyone done this before? What is the right pattern for these types of services?
Your option 2, using Mesos-DNS with static port reservation, is likely the safest (and most conservative) option.
Going forward, I suggest to also keep an eye on Mysos (MySQL on Mesos).