In my K8s Cluster, I have a prometheus stack installed via helm.
prometheus-community/kube-prometheus-stack --version 34.9.0
In the prometheus "targets", some targets giving below error while some others are working.
server returned HTTP status 502 Bad Gateway
See the image below for reference,
If we try the matrics endpoint directly, it gives metric response fine (tried in browser) on the same ports ( More over these are not manual configuration, its auto configured by the prometheus stack helm chart itself.)
I compared ServiceMonitor configurations, containers configuration of the one working and not working - all seems same.
Notes
- I have "Istio" Service mesh enabled ( istio-injection=enabled ).
- All the prometheus related resources are in same namespace "monitoring"
- Some resources in the same namespace is giving proper response. (ex: prometheus-stack-kube-prom-kubelet, prometheus-stack-kube-prom-prometheus) , while others giving 502 error.
How to fix it, and what causing the error.