Azure Kubernetes - prometheus is deployed as a part of ISTIO not showing the deployments?

Question

I have used the following configuration to setup the Istio

cat << EOF | kubectl apply -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istio-control-plane
spec:
  # Use the default profile as the base
  # More details at: https://istio.io/docs/setup/additional-setup/config-profiles/
  profile: default
  # Enable the addons that we will want to use
  addonComponents:
    grafana:
      enabled: true
    prometheus:
      enabled: true
    tracing:
      enabled: true
    kiali:
      enabled: true
  values:
    global:
      # Ensure that the Istio pods are only scheduled to run on Linux nodes
      defaultNodeSelector:
        beta.kubernetes.io/os: linux
    kiali:
      dashboard:
        auth:
          strategy: anonymous
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

and exposed the prometheus service as mentioned below

kubectl expose service prometheus --type=LoadBalancer --name=prometheus-svc --namespace istio-system
kubectl get svc prometheus-svc -n istio-system -o json
export PROMETHEUS_URL=$(kubectl get svc prometheus-svc -n istio-system  -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}"):$(kubectl get svc prometheus-svc -n istio-system -o 'jsonpath={.spec.ports[0].port}')
echo http://${PROMETHEUS_URL}
curl http://${PROMETHEUS_URL}

I have deployed an application however couldn't see the below deployments in prometheus

Answer 1

The standard prometheus installation by istio does not configure your pods to send metrics to prometheus. It just collects data from the istio resouces.

To add your pods to being scraped add the following annotations in the deployment.yml of your application:

apiVersion: apps/v1
kind: Deployment
[...]
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: true   # determines if a pod should be scraped. Set to true to enable scraping.
        prometheus.io/path: /metrics # determines the path to scrape metrics at. Defaults to /metrics.
        prometheus.io/port: 80       # determines the port to scrape metrics at. Defaults to 80.
[...]

By the way: The prometheus instance installed with istioctl should not be used for production. From docs:

[...] pass --set values.prometheus.enabled=true during installation. This built-in deployment of Prometheus is intended for new users to help them quickly getting started. However, it does not offer advanced customization, like persistence or authentication and as such should not be considered production ready.

You should setup your own prometheus and configure istio to report to it. See: Reference: https://istio.io/latest/docs/ops/integrations/prometheus/#option-1-metrics-merging

The following yaml provided by istio can be used as reference for setup of prometheus: https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/prometheus.yaml

Furthermore, if I remember correctly, installation of addons like kiali, prometheus, ... with istioctl will be removed with istio 1.8 (release date december 2020). So you might want to setup your own instances with helm anyway.