Errors when using etcdctl on Kubernetes cluster: "certificates signed by unknown authority"

Question

I have minikube running and I am trying to list the keys on my ETCD.

I downloaded the latest etcdctl client from github:
https://github.com/etcd-io/etcd/releases/download/v3.3.18/etcd-v3.3.18-linux-amd64.tar.gz

I tried to run it with the certificates from /home/myuser/.minikube/certs:

./etcdctl --ca-file /home/myuser/.minikube/certs/ca.pem 
          --key-file /home/myuser/.minikube/certs/key.pem 
          --cert-file /home/myuser/.minikube/certs/cert.pem  
          --endpoints=https://10.240.0.23:2379 get / 

I received an error:

Error: client: etcd cluster is unavailable or misconfigured; error #0: x509: certificate signed by unknown authority

error #0: x509: certificate signed by unknown authority

Did I used the correct certificates ?

I tried different certificates like that:

./etcdctl --ca-file /var/lib/minikube/certs/ca.crt 
          --key-file /var/lib/minikube/certs/apiserver-etcd-client.key 
          --cert-file /var/lib/minikube/certs/apiserver-etcd-client.crt 
          --endpoints=https://10.240.0.23:2379 get /   

I received the same error from before.

Any idea what is the problem ?

Answer 1

I needed to use the ETCDCTL_API=3 before the commands.
I saw it being used in Kubernetes the Hard Way from this Github.
The location of the certificate are in: /etc/kubernetes/pki/etcd.

The command should work like that:

ETCDCTL_API=3 ./etcdctl --endpoints=https://172.17.0.64:2379 \
                        --cacert=/etc/kubernetes/pki/etcd/ca.crt \
                        --cert=/etc/kubernetes/pki/etcd/server.crt \
                        --key=/etc/kubernetes/pki/etcd/server.key get / --prefix

I tested it and it worked for me.

Answer 2

For minikube the correct path for etcd certificates is: /var/lib/minikube/certs/etcd/ so the command will be like that:

# kubectl -n kube-system exec -it etcd-minikube -- sh -c "ETCDCTL_API=3 ETCDCTL_CACERT=/var/lib/minikube/certs/etcd/ca.crt ETCDCTL_CERT=/var/lib/minikube/certs/etcd/server.crt ETCDCTL_KEY=/var/lib/minikube/certs/etcd/server.key etcdctl endpoint health"

Answer 3

Try to execute below command: $ cat /etc/etcd.env to list CA , CERT, KEY directories(actual path).

TLS settings

ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-k8s-m1.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-k8s-m1-key.pem
ETCD_CLIENT_CERT_AUTH=true

Then you will be possible to correct use certificates.

Then run command again:

./etcdctl --endpoints https://x.x.x.x:2379 
          --ca-file=/etc/ssl/etcd/ssl/ca.pem 
          --cert-file=/etc/ssl/etcd/ssl/member-k8s-m1.pem 
          --key-file=/etc/ssl/etcd/ssl/member-k8s-m1-key.pem

More information you can find here: etcd-certificates.

Answer 4

If you want to dump all etcd entries fully prefixed but from host/outside its container, you could also issue (here for minikube/local testing):

kubectl exec -it \
  -n kube-system etcd-minikube \
  -- sh -c 'ETCDCTL_CACERT=/var/lib/minikube/certs/etcd/ca.crt \
    ETCDCTL_CERT=/var/lib/minikube/certs/etcd/server.crt \
    ETCDCTL_KEY=/var/lib/minikube/certs/etcd/server.key \
    ETCDCTL_API=3  \
    etcdctl \
      get \
      --prefix=true /'