Alertmnager: Difference between cluster listen address and advertise address

Question

I have alertmanager running on two different host machines as docker containers and both should be running as cluster. Both the machines are inside a same vpc and communicate with private ip addresses.

I need to know what is the difference between --cluster.listen-address and --cluster.advertise-address.
Correct me if I'm wrong, Specifying cluster.listen-address=:9094 means that the alertmanager will listen for cluster connections on all interfaces on port 9094. Therefore, all other alertmanager nodes on different host machine can connect to this alertmanager's ip address at port 9094 by setting cluster.peer=<ip>:9094.
But then, I do not understand what is the use of cluster.advertise-address.

The documentation doesn't give much idea about the use case of advertise and listen address.

The chosen port in the cluster.listen-address flag is the port that needs to be specified in the cluster.peer flag of the other peers.
The cluster.advertise-address flag is required if the instance doesn't have an IP address that is part of RFC 6980 with a default route.

Can someone help me know the difference between the two?

Answer 1

In a clustered system every node needs to be able to reach every other node. A node could have multiple identities, though, particularly if some form of network address translation (NAT) is in use. In the specific context of Docker, for example, a container has a container-private IP address, but outside of Docker it's usually reached through its host's IP address or DNS name and a published port.

The listen address or bind address specifies which network interfaces the process accepts inbound connections on. In Docker this almost always must be the special "all interfaces" IPv4 address 0.0.0.0 or its IPv6 equivalent ::, and uses the service's standard TCP port. The Alertmanager default 0.0.0.0:9094 will be correct here. (If it is 127.0.0.1 then the process can't be reached from outside its container. Some software will enumerate the available interfaces and separately bind to each, which will also work.)

The advertise address tells the cluster how to connect back to the node. If the entire system is running inside Docker, this could be the Docker-internal DNS name; in Kubernetes, a Service name; or in a multi-host setup, the host's name and published port. Again especially in the Docker case, the server could be listening on port 9094 but a docker run -p option could pick a different externally visible port; the advertise address configuration is the only way the rest of the cluster can know about this address.

The peers are the initial set of other nodes in the cluster, but they can come and go, and once you've connected to the cluster, a new node can generally discover all of the other current members.

In a multi-host Docker environment, you should use the default listen address; you should set the advertise address to your host's IP address and the published port; and you should list as many initial known peers as is convenient. For the first node you can't list any peers because the second node isn't running yet.