How to resolve failing or missing response / Address is not allowed from custom kubernetes api service?

Question

I am currently trying to install the Prometheus adapter into my AWS EKS Kubernetes cluster. However, the associated APIService for the Prometheus adapter is having trouble connecting to the Prometheus adapter and its status is not changing to "available"

When I run kubectl edit apiservice custom.metrics.k8s.io then I get the following info and the corresponding error message:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    meta.helm.sh/release-name: adapter-test
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2022-11-29T11:00:36Z"
  labels:
    app.kubernetes.io/component: metrics
    app.kubernetes.io/instance: adapter-test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: prometheus-adapter
    app.kubernetes.io/part-of: prometheus-adapter
    app.kubernetes.io/version: v0.10.0
    helm.sh/chart: prometheus-adapter-3.4.2
  name: v1beta1.custom.metrics.k8s.io
  resourceVersion: "644865077"
  uid: 50e1d07b-5db8-49b0-92d3-af1ec581a096
spec:
  group: custom.metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: adapter-test-prometheus-adapter
    namespace: monitoring
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2022-11-29T11:00:36Z"
    message: 'failing or missing response from https://192.168.2.20:6443/apis/custom.metrics.k8s.io/v1beta1:
      Get "https://192.168.2.20:6443/apis/custom.metrics.k8s.io/v1beta1": Address
      is not allowed'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

Versions:

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:36:36Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.15-eks-fb459a0", GitCommit:"be82fa628e60d024275efaa239bfe53a9119c2d9", GitTreeState:"clean", BuildDate:"2022-10-24T20:33:23Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

On the other hand, I can use the curl command to reach the service "https://192.168.2.20:6443/apis/custom.metrics.k8s.io/v1beta1" from a PC with kubectl without any problems: I get for the command

curl -k --header "Authorization: Bearer <<MYTOKEN>>" "https://172.20.44.186:443/apis/custom.metrics.k8s.io/v1beta1"

the answer:

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "namespaces/container_threads",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "pods/container_threads",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

In the log of the adapter, however, I can't find any entry about requests from the API service. How can I analyze the error further and eliminate the error?

Answer 1

Are you running a custom CNI (Calico, Weave, etc)?

see: https://github.com/kubernetes-sigs/prometheus-adapter/issues/268#issuecomment-729480071

tldr; you may wish to ensure that:

• hostNetwork.enabled: true
• dnsPolicy=ClusterFirstWithHostNet