I have a problem with registration of a service by a consul agent. The consul agent is listed as alive in the cluster members information, but it does not register a service or respond to queries via HTTP interface. There is an error in the log but I can not interpret it:
2015/06/16 16:09:42 [INFO] agent: Joining cluster...
2015/06/16 16:09:42 [INFO] agent: (LAN) joining: [10.10.100.226]
2015/06/16 16:09:42 [INFO] agent: (LAN) joined: 1 Err: <nil>
2015/06/16 16:09:42 [INFO] agent: Join completed. Synced with 1 initial agents
Here is the configuration of the consul agent that runs on this server:
{"data_dir":"/opt/consul","datacenter":"dc","log_level":"INFO","node_name":"app01","retry_join":["10.10.100.226"]}
And the configuration of the server. The cluster has 3 server agents.
{"client_addr":"0.0.0.0","data_dir":"/opt/consul","datacenter":"ovh-rbx","log_level":"INFO","node_name":"consul-server","server":true,"ui_dir":"/opt/consul/ui"}
I received an answer on the Consul mailing list, so I will post it here just in case someone else stumbles into the same problem:
"The log message that you pasted with the error, which says "Err: ", is actually fine. We always dump any error on that step, even if there was none. The message following that one, "Join completed", confirms that the join was successful, so this shouldn't be anything to worry about.
I noticed that you don't have any bootstrap options set in the server configuration. Bootstrapping a Consul cluster is a required step. Did you pass any bootstrap options on the command line during start? You can read about bootstrapping here: https://consul.io/docs/guides/bootstrapping.html, but basically if you don't already, you should just add "bootstrap_expect": 3 to your configuration on the server nodes."
Setting bootstrap option to number of servers, deleting the data directory and restarting the cluster solved the problem.