Cannot recover cockroachdb when a node is dead

60 views Asked by At

I deploy cockroachdb helm chart (https://github.com/cockroachdb/helm-charts/tree/master) in my kubernetes cluster. Here is my value file.

  tls:
    enabled: false
  conf:
    join:
      - myapp-cockroachdb-0.myapp-cockroachdb.myapp.svc.cluster.local:8080
      - myapp-cockroachdb-1.myapp-cockroachdb.myapp.svc.cluster.local:8080
    single-node: false
  statefulset:
    replicas: 2

It worked. But when I restarted the 2 nodes at the same time, it failed because each node tries to connect to each other which is not ready.

So I am trying to start only myapp-cockroachdb-0. I changed the value file to

  tls:
    enabled: false
  conf:
    join:
      - myapp-cockroachdb-0.myapp-cockroachdb.myapp.svc.cluster.local:8080
    single-node: false
  statefulset:
    replicas: 1

But myapp-cockroachdb-0 still try to connect myapp-cockroachdb-1. I read the documentation, it suggests that I run cockroach node decommission command but I cannot because the node has not started yet.

0

There are 0 answers