When querying Datasource from grafana getting 403 for a few of the metrics.
curl --location --request GET 'https://xxx.xxx.xxx/api/datasources/proxy/1/api/v1/query?query=sum(kube_pod_container_status_restarts_total%7Bnamespace%3D%22default%22%2C%20container%3D~%22al-agent-container%22%2C%20pod%3D%22al-agent-container-hlrz2%22%7D)&time=1607489911' \
> --header 'Accept: application/json' \
> --header 'Content-Type: application/json; charset=UTF-8' \
> --header 'Authorization: Bearer xxx' \
> --data-raw ''
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>
When Querying from the Prometheus server endpoint with same query it resolves with 200:
curl --location --request GET 'http://kube-system-prometheus-server.kube-system.svc.cluster.local/api/v1/query_range?query=sum(kube_pod_container_resource_requests_cpu_cores%7Bnamespace%3D%22default%22%2C%20pod%3D~%22al-agent-container-hlrz2%22%7D)&start=1607488035&end=1607489835&step=15' \
> --header 'Accept: application/json' \
> --header 'Content-Type: application/json; charset=UTF-8'
{"status":"success","data":{"resultType":"matrix","result":[]}}
Grafana Version: v6.6.2 (3fa63cfc34)
Prometheus does not have any authentication in place, This issue is facing only for few metrics not for all.
When tested the data source from its configuration section. it shows it's all working.
does anyone have any idea on this?
I have found this issue,
I had WAF enabled in the entry LB, I have removed and checked that it does work. WAF is considering this as SQL injection and hence blocking it.