Zeebe metrics monitoring using external prometheus

Question

I deployed a zeebe-cluster in kubernetes from the github repositories and I am trying to connect them with a prometheus that I have already deployed. I have been reading the documentation and I saw that zeebe already generates metrics for prometheus on port 9600.

My question is when I configure the prometheus configmap, which service should I set as target.

     - job_name: zeebe
       scrape_interval: 15s
       metrics_path: /metrics
       scheme: http
       static_configs:
       - targets:
         - localhost: 9600 

When I check the services, I observe that there are 2 that are generated from the zeebe-cluster one called RELEASE-zeebe and another RELEASE-zeebe-gateway

zeebe-cluster-zeebe           ClusterIP   172.20.41.249   <none>        9600/TCP,26502/TCP,26501/TCP

zeebe-cluster-zeebe-gateway   ClusterIP   172.20.93.148   <none>        9600/TCP,26500/TCP 

When i did a “kubectl port-forward svc/xxxx” through port 9600 (localhost:9600/metrics) to the two services I see that they generate different metrics.

According to what I have understood from the documentation. The zeebe-gateway is a gateway to the zeebe-cluster brokers. I would suppose that I should put the gateway service. And, Do I need to do any extra configuration on the zeebe-cluster?

Answer 1

I would suggest you deploy the prometheus-operator. You can find resources how we do it on our Zeebe benchmark cluster here.

After that you can use the helm charts (which I assume you already do) and enable the service monitors, this allows to scrape the metrics from the services. Basically it is done via labels.

So enable prometheus.servicemonitor.enabled, see zeebe-cluster helm chart README for more information about that.

If you want to understand better how it works with services and with the service monitor you can check the prometheus operator getting started guide