How to automatically scrape all Docker instances from Kubernetes with Prometheus?

Question

I've successfully installed Prometheus in Google Container Engine and I have these targets up:

• kubernetes-apiservers
• kubernetes-cadvisor
• kubernetes-nodes

Now I would like to scrape Nginx stats from each of the Docker containers inside this Kubernetes cluster (which seems like a sensible thing to do).

But how can I make Prometheus automatically pull the metrics from all the Nginx instances running in all of the Docker containers?

From my research so far, the answer involves kubernetes_sd_config but I simply could not find enough documentation on how to put the pieces together.

Thank you!

Edit: This is not about exposing the Nginx stats. This is just about scraping any stats that are exposed by all Docker containers.

Answer 1

You are correct that you need to use the kubernetes_sd_config directive. Before continuing let me just say that what you should be asking is "Automatically scape all pods from Kubernetes". This is because a pod is considered the lowest unit of scale in Kubernetes. Regardless it is clear what you are trying to do.

So the kubernetes_sd_config can be used to discover all pods with a given tag like so:

- job_name: 'some-app'
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      regex: python-app
      action: keep

The source label [__meta_kubernetes_pod_label_app] is basically using the Kubernetes api to look at app pods that have a label of 'app' and whose value is captured by the regex expression, given on the line below (in this case, matching 'python-app').

Hope that helps. You can follow blog post here for more detail. Also for more information about kubernetes_sd_config check out docs here.

Note: it is worth mentioning that kubernetes_sd_config is still in beta. Thus breaking changes to configuration may occur in future releases.