Prometheus adapter return empty custom metrics

Question

Having configured metric rule in the prometheus-adapter, I can see resources as follows.

kubectl get --raw="/apis/custom.metrics.k8s.io/v1beta1" | jq .
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "pods/istio_ai_retry_per_second",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "namespaces/istio_ai_retry_per_second",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

But when I try to get metrics values. The items are empty.

kubectl get --raw="/apis/custom.metrics.k8s.io/v1beta1/namespaces/aiservice-test/pods/*/istio_ai_retry_per_second" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/aiservice-test/pods/%2A/istio_ai_retry_per_second"
  },
  "items": []
}

This is the rule,

 10   - seriesQuery: 'istio_ai_retry_count{destination_workload="cpuporncheck-default-k8s",reporter="destination"}'
 11     resources:
 12       overrides:
 13         destination_workload: {resource: "pod"}
 14         destination_workload_namespace: {resource: "namespace"}
 15     name:
 16       matches: "^(.*)_count"
 17       as: "${1}_per_second"
 18     metricsQuery: "sum(rate(istio_ai_retry_count{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)"

Can anyone help me to check what's wrong with it?

Answer 1

OP confirmed that the problem was with rules.

Changeing pod to deployment and api access path to sth like:

 /apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace_name}/deployment/{deployment_name}/{metric}

resolved the issue.