Prometheus: Monitor all services without creating ServiceMonitor for each service?

Question

I'm using this prometheus helm chart.

I was wondering if it is possible to setup the prometheus operator to automatically monitor every service in the cluster or namespace without having to create a ServiceMonitor for every service.

With the current setup, when I want to monitor a service, I have to create a ServiceMonitor with the label release: prometheus.

Edit:

Service with monitoring: "true" label

apiVersion: v1
kind: Service
metadata:
  name: issue-manager-service
  labels:
    app: issue-manager-app
    monitoring: "true"
spec:
  selector:
    app: issue-manager-app
  ports:
    - protocol: TCP
      name: http
      port: 80
      targetPort: 7200

"Catch-All" Servicemonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: service-monitor-scraper
  labels:
    release: prometheus
spec:
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
  jobLabel: monitoring
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      monitoring: "true"

Answer 1

Only if you have a common label on all services

# for example:
org: "my-company"
# or
monitoring: "true"
# or 
app.kubernetes.io/managed-by: "Helm"  # <- in most cases this represents all

Then you define a single, cross-namespace ServiceMonitor, that covers all labeled services:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: common-monitor
  namespace: monitoring
spec:
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
  jobLabel: monitoring
  namespaceSelector:
    any: true  # <- important
  selector:
    matchLabels:
      monitoring: "true"  # <- should match what you've chosen as common

Then to make sure this ServiceMonitor is discovered by the Prometheus Operator you either:

• Declare this ServiceMonitor via the built-in operator template: https://github.com/prometheus-community/helm-charts/blob/4164ad5fdb6a977f1aba7b65f4e65582d3081528/charts/kube-prometheus-stack/values.yaml#L2008
• Or, declare a serviceMonitorSelector that points to your ServiceMonitor https://github.com/prometheus-community/helm-charts/blob/4164ad5fdb6a977f1aba7b65f4e65582d3081528/charts/kube-prometheus-stack/values.yaml#L1760

This additional explicit linkage between Prometheus Operator and ServiceMonitor is done intentionally - in this way, if you have 2 Prometheus instances on your cluster (say Infra and Product) you can separate which Prometheus will get which Pods to its scraping config.

From your question, it sounds like you already have a serviceMonitorSelector based on release: prometheus label - try adding that on your catch-all ServiceMonitor as well.