I am following the Coreos guide for Enabling HTTPS in an existing etcd cluster. Link to Documentation
I have two questions:
1) After reconfiguring the Peer URL to HTTPS using the following command as stated in the documentation:
etcdctl member list | awk -F'[: =]' '{print "etcdctl member update "$1" https:"$7":"$8}'
The output is the same as the documentation, but what is the output I am suppose to after running:
etcdctl member list
I am suppose to see the peerUrls updated to HTTPs?
2) I went ahead and configured documentation and change the etcd client URLs. After doing that the cluster completely stops communicating:
etcd2[5063]: 5ebdc721c084a4b1 is starting a new election at term 20548
etcd2[5063]: 5ebdc721c084a4b1 became candidate at term 20549
etcd2[5063]: 5ebdc721c084a4b1 received vote from 5ebdc721c084a4b1 at term 20549
etcd2[5063]: 5ebdc721c084a4b1 [logterm: 20478, index: 6405417] sent vote request to d5df37b45e3cb90f at term 20549
etcd2[5063]: 5ebdc721c084a4b1 [logterm: 20478, index: 6405417] sent vote request to f3aee5692d89a2a3 at term 20549
etcd2[5063]: 5ebdc721c084a4b1 [logterm: 20478, index: 6405417] sent vote request to fb362473ced21e89 at term 20549
etcd2[5063]: the connection to peer d5df37b45e3cb90f is unhealthy
etcd2[5063]: the connection to peer f3aee5692d89a2a3 is unhealthy
etcd2[5063]: the connection to peer fb362473ced21e89 is unhealthy
And when I running the following command to debug:
sudo etcdctl --ca-file /etc/ssl/etcd/ca.pem --cert-file /etc/ssl/etcd/server.pem --key-file /etc/ssl/etcd/server-key.pem member list
I get the following output
Failed to get leader: client: etcd cluster is unavailable or misconfigured
And when I run curl as in the documentation its gives me the correct input
curl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/server.pem --key /etc/ssl/etcd/server-key.pem https://172.16.0.2:2379/v2/stats/self
{"name":"coreos0","id":"5ebdc721c084a4b1","state":"StateFollower","startTime":"2017-03-21T11:33:13.964177689+03:00","leaderInfo":{"leader":"fb362473ced21e89","uptime":"13m37.308602575s","startTime":"2017-03-21T11:33:14.480109854+03:00"},"recvAppendRequestCnt":33,"sendAppendRequestCnt":0}
My Configuration
/run/systemd/system/etcd2.service.d/20-cloudinit.conf
[Service]
Environment="ETCD_ADVERTISE_CLIENT_URLS=http://172.16.0.2:2379"
Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.16.0.2:2380"
Environment="ETCD_INITIAL_CLUSTER=coreos1=http://172.16.0.4:2380,coreos2=http://172.16.0.5:2380,coreos0=http://172.16.0.2:2380"
Environment="ETCD_INITIAL_CLUSTER_STATE=new"
Environment="ETCD_INITIAL_CLUSTER_TOKEN=cluster1"
Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379"
Environment="ETCD_LISTEN_PEER_URLS=http://172.16.0.2:2380"
Environment="ETCD_NAME=coreos0"
/etc/systemd/system/etcd2.service.d/25-insecure_localhost.conf
[Service]
Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://127.0.0.1:4001"
/etc/systemd/system/etcd2.service.d/30-certs.conf
[Service]
Environment="ETCD_CERT_FILE=/etc/ssl/etcd/server.pem"
Environment="ETCD_KEY_FILE=/etc/ssl/etcd/server-key.pem"
Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
Environment="ETCD_CLIENT_CERT_AUTH=true"
Environment="ETCD_PEER_CERT_FILE=/etc/ssl/etcd/server.pem"
Environment="ETCD_PEER_KEY_FILE=/etc/ssl/etcd/server-key.pem"
Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
/etc/systemd/system/etcd2.service.d/40-tls.conf
[Service]
Environment="ETCD_ADVERTISE_CLIENT_URLS=https://172.16.0.2:2379"
Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379,http://127.0.0.1:4001"
Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
Thank you in advance
There was a bug in cfssl that cause the peer certificates to be generated wrong. After regeneration was created with echo, the issue resvolbed itself.