Why Istio does not generate the following metrics for HTTP, HTTP/2, and GRPC traffic?
My Istio installation:
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update
kubectl create namespace istio-system
helm install istio-base istio/base -n istio-system
helm install istiod istio/istiod -n istio-system --wait
$ istioctl version
client version: 1.8.1
control plane version: 1.18.0
data plane version: 1.18.0 (10 proxies)
$ kubectl version --short
Client Version: v1.27.2
Kustomize Version: v5.0.1
Server Version: v1.27.2
$ helm version --short
v3.12.0+gc9f554d
$ helm list
istio-base istio-system 1 2023-06-13 08:39:32.349599821 +0300 MSK deployed base-1.18.0 1.18.0
istiod istio-system 1 2023-06-13 08:39:34.034866006 +0300 MSK deployed istiod-1.18.0 1.18.0
In documentation (https://istio.io/latest/docs/reference/config/metrics/) wrote:
For HTTP, HTTP/2, and GRPC traffic, Istio generates the following metrics:
Request Count (istio_requests_total): This is a COUNTER incremented for every request handled by an Istio proxy.
Request Duration (istio_request_duration_milliseconds): This is a DISTRIBUTION which measures the duration of requests.
Request Size (istio_request_bytes): This is a DISTRIBUTION which measures HTTP request body sizes.
Response Size (istio_response_bytes): This is a DISTRIBUTION which measures HTTP response body sizes.
gRPC Request Message Count (istio_request_messages_total): This is a COUNTER incremented for every gRPC message sent from a client.
gRPC Response Message Count (istio_response_messages_total): This is a COUNTER incremented for every gRPC message sent from a server.
But my Prometneus contains only TCP metrics:
istio_tcp_sent_bytes_total
istio_tcp_received_bytes_total
istio_tcp_connections_opened_total
istio_tcp_connections_closed_total
In my Prometheus config follow additional ScrapeConfigs:
- job_name: 'istiod'
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- istio-system
relabel_configs:
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: istiod;http-monitoring
- job_name: 'envoy-stats'
metrics_path: /stats/prometheus
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_port_name]
action: keep
regex: '.*-envoy-prom'
Even despite the fact that I deployed echo-grpc
according to this instruction https://istio.io/latest/blog/2021/proxyless-grpc/ these metrics are still missing in Prometheus.
Please tell me what is my mistake?