Here is a query that is working well in Prometheus.
rate(starlette_requests_total{method="POST", path_template="/predict"}[1m])
Prometheus stack is installed via a helm chart:
kubectl create namespace monitoring && \
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && \
helm repo update && \
helm install -n monitoring prometheus-stack prometheus-community/kube-prometheus-stack
I tried to make the rule as follows:
helm install -n monitoring myprom-adapter prometheus-community/prometheus-adapter -f values.yaml
where values.yaml contains:
rules:
custom:
- seriesQuery: 'starlette_requests_total'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: "starlette_requests_total"
as: "prediction_requests_rate_1m"
metricsQuery: rate(starlette_requests_total{method="POST", path_template="/predict"}[1m])
The metric doesn't show up in the custom metric:
# kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[]}
I'm running these on a Minikube cluster (v1.26.0
) with these helm chart versions: prometheus-adapter-3.3.1
and kube-prometheus-stack-37.2.0
What would a corresponding Prometheus Adapter Rule look like to get this exposed in the custom-metrics API for HPA to use?