How to use regex to reduce cardinality of istio metrics which caused istio side car crash

Question

I am using istio as my kubernetes service mesh. And I want to use metrics such as istio_requests_total as my SLI. I need to add a customized label "request_url_path" into those metrics, whose value is set to the real request url(request.url).

However, after I apply those customization at my metrics, during my load tests, istio-proxy sidecar keeps crashing because of OOMKilled.

After investigation, I think it was cause by the dynamic part inside the request.url(such as some GUID, or random numbers), for example: /api/user/27ee8e1c8bcedd1becerqfe

I want to remove those dynamic parts inside the url and only apply those static part, below is my test code for applying a customized metrics with telemetry v2. It checks whether the url path include "login", or "logout", or "api/user", if yes, use "login", or "logout", or "api/user" as the label's value, otherwise, use "unknown" as the label's value.

However, below configuration doesn't work.

It seems the find function(for Common Expression Language regex) is not defined in istio.

Could you provide some suggestions on how can I solve my problem? Thanks a lot!!

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: add-tags
  namespace: istio-system
spec:
  metrics:
    - providers:
        - name: prometheus
      overrides:
        - match:
            metric: REQUEST_COUNT
          tagOverrides:
            request_url_path:
              value: request.url_path.matches('^/(login|logout|api/user).*$')?request.url_path.find('/(login|logout|/api/user)'):"unknown"

Answer 1

We ended up with using lua envoyfilter to remove the Guid part inside the request path and add this as a http request header. Then add this extra header as a istio metrics dimension.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: request-add-uri-headers-filter
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          inlineCode: |
            function envoy_on_request(request_handle)
              -- Log information about the request
              local x = "%x"
              local t = { x:rep(8), x:rep(4), x:rep(4), x:rep(4), x:rep(12) }
              local pattern_guid = table.concat(t, '%-')
              local pattern_uri = '([^%?]+).*'
              local uripath = string.match(request_handle:headers():get(":path"),pattern_uri)
              local strippedpath = string.gsub(uripath,pattern_guid,'{GUID}')
              request_handle:logInfo("Path: "..strippedpath)
              request_handle:headers():add("request_path_header", strippedpath)
            end