I use Istio 1.8 for service mesh and Prometheus to collect metrics from sidecards. Currently these metrics are been provided by sidecards:
istio_request_bytes_bucket
istio_request_duration_milliseconds_bucket
istio_requests_total
envoy_cluster_upstream_cx_connect_ms_bucket
istio_request_messages_total
istio_response_messages_total
envoy_cluster_upstream_cx_length_ms_bucket
istio_response_bytes_bucket
istio_request_bytes_sum
istio_request_bytes_count
This amount of metrics use lots of network bandwidth. (We have around 5k pods)
All we need for now are istio_requests_total
and istio_request_duration_milliseconds_bucket
only from Inbound.
I know how to remove labels by EnvoyFilter
but I was unable to find documentation for removing a metric.
Here you can find the info regarding customizing Istio (1.8) metrics :
https://istio.io/v1.8/docs/tasks/observability/metrics/customize-metrics/