Troubleshooting a ravendb deployment on kubernetes

Question

I had deployed ravendb on a 3 node cluster on google cloud. However it's not accessible from the browser. Here is the procedure and the configuration I had followed. Could you please help me troubleshoot the service and deployment. When I run the get pods and get svc commands the pods and services are running but the db isn't accessible from the browser.

Procedure followed:

I suggest you first run the setup wizard on your local dev machine and get the Let's Encrypt certificate. Just use 127.0.0.1:8080 as the IP, it's not important at the moment.
(Even better will be to get your own domain + certificate for production use)

You need to convert both the pfx file and the license.json file to base64, In c# for example:
Convert.ToBase64String(File.ReadAllBytes(@"C:\work\certs\cluster.server.certificate.iftah.pfx"))

Convert.ToBase64String(File.ReadAllBytes(@"C:\work\license.json"))

1. Create a GKE standard cluster with 3 nodes, no special settings. Let's call it raven-cluster
2. Install gcloud and kubectl (follow the getting started guide: https://cloud.google.com/kubernetes-engine/docs/quickstart)
run:
3. > gcloud container clusters get-credentials raven-cluster
4. > kubectl create clusterrolebinding my-cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)

Now you're ready to deploy. 
Edit the cluster.yaml file to include the base64 certificate (name: raven-ssl)
Edit the license.secret.yaml file to include the base64 license (name: ravendb-license)

4a) kubectl label node role=ingress-controller --all
5. kubectl create -f license.secret.yaml
6. kubectl create -f haproxy.yaml
7. kubectl create -f cluster.yaml
9. kubectl get pod
8. kubectl get svc

ravendb spec YAML:

apiVersion: v1
items:
- apiVersion: v1
  data:
    raven-0: "{\r\n  \"Setup.Mode\": \"None\",\r\n  \"DataDir\": \"/data/RavenData\",\r\n
      \ \"Security.Certificate.Path\": \"/ssl/ssl\",\r\n  \"ServerUrl\": \"https://0.0.0.0\",\r\n
      \ \"ServerUrl.Tcp\": \"tcp://0.0.0.0:38888\",\r\n  \"PublicServerUrl\": \"https://a.tej-test001.ravendb.community\",\r\n
      \ \"PublicServerUrl.Tcp\": \"tcp://tcp-a.tej-test001.ravendb.community:443\",\r\n
      \ \"License.Path\": \"/license/license.json\",\r\n  \"License.Eula.Accepted\":
      \"true\",\r\n  \"License.CanActivate\": \"false\",\r\n  \"License.CanForceUpdate\":
      \"false\",\r\n  \"Server.AllowedDestinations\": \"Azure\",\r\n}"
    raven-1: "{\r\n  \"Setup.Mode\": \"None\",\r\n  \"DataDir\": \"/data/RavenData\",\r\n
      \ \"Security.Certificate.Path\": \"/ssl/ssl\",\r\n  \"ServerUrl\": \"https://0.0.0.0\",\r\n
      \ \"ServerUrl.Tcp\": \"tcp://0.0.0.0:38888\",\r\n  \"PublicServerUrl\": \"https://b.tej-test001.ravendb.community\",\r\n
      \ \"PublicServerUrl.Tcp\": \"tcp://tcp-b.tej-test001.ravendb.community:443\",\r\n
      \ \"License.Path\": \"/license/license.json\",\r\n  \"License.Eula.Accepted\":
      \"true\",\r\n  \"License.CanActivate\": \"false\",\r\n  \"License.CanForceUpdate\":
      \"false\",\r\n  \"Server.AllowedDestinations\": \"Azure\",\r\n}"
    raven-2: "{\r\n  \"Setup.Mode\": \"None\",\r\n  \"DataDir\": \"/data/RavenData\",\r\n
      \ \"Security.Certificate.Path\": \"/ssl/ssl\",\r\n  \"ServerUrl\": \"https://0.0.0.0\",\r\n
      \ \"ServerUrl.Tcp\": \"tcp://0.0.0.0:38888\",\r\n  \"PublicServerUrl\": \"https://c.tej-test001.ravendb.community\",\r\n
      \ \"PublicServerUrl.Tcp\": \"tcp://tcp-c.tej-test001.ravendb.community:443\",\r\n
      \ \"License.Path\": \"/license/license.json\",\r\n  \"License.Eula.Accepted\":
      \"true\",\r\n  \"License.CanActivate\": \"false\",\r\n  \"License.CanForceUpdate\":
      \"false\",\r\n  \"Server.AllowedDestinations\": \"Azure\",\r\n}"
  kind: ConfigMap
  metadata:
    labels:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
    name: raven-settings
    namespace: default
- apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    labels:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
    name: raven
    namespace: default
  spec:
    podManagementPolicy: OrderedReady
    replicas: 3
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: ravendb
        cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
    serviceName: raven
    template:
      metadata:
        labels:
          app: ravendb
          cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
      spec:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: cluster
                  operator: In
                  values:
                  - ee632d20-0a5f-40e4-a84a-5294da32d6d5
              topologyKey: kubernetes.io/hostname
        containers:
        - command:
          - /bin/sh
          - -c
          - /opt/RavenDB/Server/Raven.Server --config-path /config/$HOSTNAME
          image: ravendb/ravendb:latest
          imagePullPolicy: Always
          name: ravendb
          ports:
          - containerPort: 443
            name: http-api
            protocol: TCP
          - containerPort: 38888
            name: tcp-server
            protocol: TCP
          - containerPort: 161
            name: snmp
            protocol: TCP
          resources:
            limits:
              cpu: 256m
              memory: 1900Mi
            requests:
              cpu: 256m
              memory: 1900Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /data
            name: data
          - mountPath: /ssl
            name: ssl
          - mountPath: /license
            name: license
          - mountPath: /config
            name: config
        dnsPolicy: ClusterFirst
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        terminationGracePeriodSeconds: 120
        volumes:
        - name: ssl
          secret:
            defaultMode: 420
            secretName: raven-ssl
        - configMap:
            defaultMode: 420
            name: raven-settings
          name: config
        - name: license
          secret:
            defaultMode: 420
            secretName: ravendb-license
    updateStrategy:
      rollingUpdate:
        partition: 0
      type: RollingUpdate
    volumeClaimTemplates:
    - metadata:
        labels:
          app: ravendb
          cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
- apiVersion: extensions/v1beta1
  kind: Ingress  
  metadata:
    annotations:
      ingress.kubernetes.io/ssl-passthrough: "true"
      kubernetes.io/ingress.class: "haproxy"
    labels:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
    name: raven
    namespace: default
  spec:
    rules:
    - host: a.tej-test001.ravendb.community
      http:
        paths:
        - backend:
            serviceName: raven-0
            servicePort: 443
          path: /
    - host: tcp-a.tej-test001.ravendb.community
      http:
        paths:
        - backend:
            serviceName: raven-0
            servicePort: 38888
          path: /
    - host: b.tej-test001.ravendb.community
      http:
        paths:
        - backend:
            serviceName: raven-1
            servicePort: 443
          path: /
    - host: tcp-b.tej-test001.ravendb.community
      http:
        paths:
        - backend:
            serviceName: raven-1
            servicePort: 38888
          path: /
    - host: c.tej-test001.ravendb.community
      http:
        paths:
        - backend:
            serviceName: raven-2
            servicePort: 443
          path: /
    - host: tcp-c.tej-test001.ravendb.community
      http:
        paths:
        - backend:
            serviceName: raven-2
            servicePort: 38888
          path: /
- apiVersion: v1
  data:
    ssl: dfdjfdkljfdkjdkjd;kfjdkfjdklfj
  kind: Secret
  metadata:
    labels:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
    name: raven-ssl
    namespace: default
  type: Opaque
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
    name: raven
    namespace: default
  spec:
    clusterIP: None
    ports:
    - name: http-api
      port: 443
      protocol: TCP
      targetPort: 443
    - name: tcp-server
      port: 38888
      protocol: TCP
      targetPort: 38888
    - name: snmp
      port: 161
      protocol: TCP
      targetPort: 161
    selector:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
      node: "0"
    name: raven-0
    namespace: default
  spec:
    ports:
    - name: http-api
      port: 443
      protocol: TCP
      targetPort: 443
    - name: tcp-server
      port: 38888
      protocol: TCP
      targetPort: 38888
    - name: snmp
      port: 161
      protocol: TCP
      targetPort: 161
    selector:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
      statefulset.kubernetes.io/pod-name: raven-0
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
      node: "1"
    name: raven-1
    namespace: default
  spec:
    ports:
    - name: http-api
      port: 443
      protocol: TCP
      targetPort: 443
    - name: tcp-server
      port: 38888
      protocol: TCP
      targetPort: 38888
    - name: snmp
      port: 161
      protocol: TCP
      targetPort: 161
    selector:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
      statefulset.kubernetes.io/pod-name: raven-1
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: v1
  kind: Service
  metadata:
    labels:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
      node: "2"
    name: raven-2
    namespace: default
  spec:
    ports:
    - name: http-api
      port: 443
      protocol: TCP
      targetPort: 443
    - name: tcp-server
      port: 38888
      protocol: TCP
      targetPort: 38888
    - name: snmp
      port: 161
      protocol: TCP
      targetPort: 161
    selector:
      app: ravendb
      cluster: ee632d20-0a5f-40e4-a84a-5294da32d6d5
      statefulset.kubernetes.io/pod-name: raven-2
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
kind: List

haproxy spec yaml:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-controller
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: ingress-controller
  namespace: default
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      - create
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-controller
subjects:
  - kind: ServiceAccount
    name: ingress-controller
    namespace: default
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: ingress-controller
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-controller
subjects:
  - kind: ServiceAccount
    name: ingress-controller
    namespace: default
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ingress-controller
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: ingress-default-backend
  name: ingress-default-backend
  namespace: default
spec:
  selector:
    matchLabels:
      run: ingress-default-backend
  template:
    metadata:
      labels:
        run: ingress-default-backend
    spec:
      containers:
      - name: ingress-default-backend
        image: gcr.io/google_containers/defaultbackend:1.0
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-default-backend
  namespace: default
spec:
  ports:
  - port: 8080
  selector:
    run: ingress-default-backend
---
apiVersion: v1
data:
  backend-server-slots-increment: "4"
  dynamic-scaling: "true"
kind: ConfigMap
metadata:
  name: haproxy-ingress
  namespace: default
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress
spec:
  selector:
    matchLabels:
      run: haproxy-ingress
  template:
    metadata:
      labels:
        run: haproxy-ingress
    spec:
      serviceAccountName: ingress-controller
      containers:
      - name: haproxy-ingress
        image: quay.io/jcmoraisjr/haproxy-ingress
        args:
        - --default-backend-service=$(POD_NAMESPACE)/ingress-default-backend
        - --configmap=$(POD_NAMESPACE)/haproxy-ingress
        - --reload-strategy=reusesocket
        ports:
        - name: https
          containerPort: 443
        - name: stat
          containerPort: 1936
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10253
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress
  namespace: default
spec:
  type: LoadBalancer
  ports:
  - name: https
    port: 443
  - name: stat
    port: 1936
  selector:
    run: haproxy-ingress

Answer 1

The instructions you have are partial. You didn't update the DNS records.

8. kubectl get pod
9. kubectl get svc
10. Write down the EXTERNAL-IP field of the haproxy-ingress service. It can take a couple of minutes until the IP is allocated. You will need to use it to update the DNS record of your domain. Go to customers.ravendb.net and edit the DNS record to the new external IP you got. (or if you have your own domain, do this with your domain provider)
11. When all the raven pods are ready, you can go to the browser and access the cluster.