Prometheus Adapter Not Able to Pull Custom Metrics from Prometheus

Question

I am trying to configure the custom metrics to trigger the HPA based on http_server_requests_second, however I am not able to successfully pull the data of custom metrics from prometheus adapter to prometheus. Although the custom data shows up correctly in the prometheus.

I have used the helm approach to install the prometheus operator and prometheus adapter from the prometheus community itself and used additionalscrapeconfig to pull the metric for my demo application.

Custom Metric Data Showing up Correctly in Prometheus

Below is the snippet of the prometheus scrape config of my application

- job_name: "appops-springboot-hpa-demo-app"
  metrics_path: /actuator/prometheus
  static_configs:
    - targets: ["springboot-hpa-app.default.svc.cluster.local:8080"]

Below are different approaches I tried to change in the configmap of the prometheus adapter, however I am continuously getting 404 while pulling the information in prometheus adapter when the custom metric api is invoked. I have attached the 404 log at end of this post.

Please note, when I invoke custom metric API, it does pull all other default metrics comes with the helm installation itself however only my custom metrics data is not showing which means at least the connectivity is fine between prometheus adapter to prometheus.

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

Approach 1 : (Didnt work)

- seriesQuery: 'http_server_requests_seconds_count{namespace !="",pod!="",uri="/requests"}'
      seriesFilters: []
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      name:
        matches: ""
        as: ""
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)'

Approach 2: (Didnt work)

custom:
    - seriesQuery: '{__name__=~"^http_server_requests_seconds_.*"}' 
      resources:
        overrides:
          kubernetes_namespace:
            resource: namespace
          kubernetes_pod_name:
            resource: pod
      name:
        matches: "^http_server_requests_seconds_count(.*)" 
        as: "http_server_requests_seconds_count_sum" 
      metricsQuery: sum ( <<.Series>> { <<.LabelMatchers>>,uri=~ "/requests.*" } ) by ( <<.GroupBy>> )

Approach 3: (Didnt work)

    - seriesQuery: 'http_server_requests_seconds_count{job="appops-springboot-hpa-demo-app", method="GET", status="200", uri="/requests"}' 
      resources:
        overrides:
          kubernetes_namespace:
            resource: namespace
          kubernetes_pod_name:
            resource: pod
      name:
        matches: http_server_requests_seconds_count 
        as: "http_server_requests_seconds_count_sum" 
      metricsQuery: rate(http_server_requests_seconds_count{job="appops-springboot-hpa-demo-app", method="GET", status="200", uri="/requests",<<.LabelMatchers>>}[5m]) 

Output of the custom Metrics API for the expected data:

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/*/http_server_requests_seconds_count | jq .

Error from server (NotFound): the server could not find the requested resource

Logs from Prometheus Adapter deployment :

I0211 00:58:38.846373       1 httplog.go:104] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/ops-poc/pods/%2A/http_server_requests_seconds_count?labelSelector=app%3Dspringboot-custom-hpa" latency="15.099215ms" userAgent="kube-controller-manager/v1.22.13 (linux/amd64) kubernetes/a43c090/system:serviceaccount:kube-system:horizontal-pod-autoscaler" audit-ID="351e31aa-c8c9-4558-bb33-f265638ef678" srcIP="10.42.15.0:57042" resp=404

Answer 1

your metric doesnt have any reference to namespace or pod so you should not use

resources: overrides: kubernetes_namespace: resource: namespace kubernetes_pod_name: resource: pod you need to make the configuration match the metric

Discovery, which specifies how the adapter should find all Prometheus metrics for this rule.

Association, which specifies how the adapter should determine which Kubernetes resources a particular metric is associated with.

Naming, which specifies how the adapter should expose the metric in the custom metrics API.

Querying, which specifies how a request for a particular metric on one or more Kubernetes objects should be turned into a query to Prometheus.

https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config.md