Consul set up without docker for production use

Question

I am doing a POC on Consul for supporting service discovery and multiple microservice versions. Consul clients and server cluster(3 servers) are set up on Linux VMs. I followed the documentation at Consul and the set up is successful.

Here is my doubt. My set up is completely on VMs. I've added a service definition using HTTP API. The same service is running on two nodes. The services are correctly registered:

curl http://localhost:8600/v1/catalog/service/my-service

gives me the two node details.

When I do a DNS query:

dig @127.0.0.1 -p 8600 my-service.service.consul

I am able to see the expected results with the node which hosts the service. But I cannot ping the service since the service name is not resolved.

ping -c4 my-service or ping -c4 my-service.service.consul

ping: unknown host.

If I enter a mapping for my-service in /etc/hosts file, I can ping this, only from the same VM. I won't be able to ping this from another VM on the same LAN or WAN. The default port for DNS is 53. Consul DNS interface listens to 8600. I cannot use Docker for DNS forwarding. Is it possible I missed something here? Can consul DNS query work without Docker/dnsmasq or iptables updates? To be clear, here is what I would like to have as the end result:

ping my-service

This needs to ping the nodes I have configured, in a round robin fashion.

Please bear with me if this question is basic, and I've gone through each of the consul related questions in SO.

Also gone through this and this and these too says I need to do extra set up.

Answer 1

Wait! Please don't do this!

DO. NOT. RUN. CONSUL. AS. ROOT.

Please. You can, but don't. Instead do the following:

1. Run a caching or forwarding DNS server on your VMs. I'm bias toward dnsmasq because of its simplicity and stability in the common case.
2. Configure dnsmasq to forward the TLD .consul to the consul agent listening on 127.0.0.1:8600 (the default).
3. Update your /etc/resolv.conf file to point to 127.0.0.1 as your nameserver.

There are a few ways of doing this, and the official docs have a write up that is worth looking into:

https://www.consul.io/docs/guides/forwarding.html

That should get you started.

Answer 2

This can be a pretty complicated topic, but the simplest way is to change consul to bind to port 53 using the ports directive, and add some recursers to the consul config can pass real DNS requests on to a host that has full DNS functionality. Something like these bits:

{ "recursors": [ "8.8.8.8", "8.8.4.4" ], "ports": { "dns": 53 } }

Then modify your system to use the consul server for dns with a nameserver entry in /etc/resolve.conf. Depending on your OS, you might be able to use a port in the resolv.conf file, and avoid having to deal with Consul needing root to bind to port 53.

In a more complicated scenario, I know many people that use unbound or bind to do split DNS and essentially do the opposite, routing the .consul domain to the consul cluster on a non-privileged port at the org level of their DNS infrastructure.