Otel Processor metric filter is not working as expected

144 views Asked by At

I have the following configuration for my OpenTelemetry (otel) filter processor. Within the receiver configuration, kubelet is scraping node metrics, and the filter is intended to selectively include or exclude metrics based on the filter configuration.

Here's the scrape configuration job in the receiver:

job_name: kubelet
scrape_interval: 1m
scheme: https
tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
  - role: node
    api_server: null
relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - target_label: __address__
    replacement: kubernetes.default.svc:443
  - source_labels: [__meta_kubernetes_node_name]
    action: replace
    target_label: __metrics_path__
    regex: (ip-.*)
    replacement: /api/v1/nodes/$${1}/proxy/metrics


In the filter configuration:

filter/metrics:
  error_mode: ignore
  metrics:
    include:
      match_type: regexp
      metric_names:
        - otelcol_receiver_accepted_metric_points_total$
        - jvm_memory_used_bytes$
        - jvm_threads_states_threads$
        - container_cpu_usage_seconds_total$
        - kubelet_runtime_operations_duration_seconds_bucket$

However, I'm encountering an issue. While I expect to see the metric name 'kubelet_runtime_operations_duration_seconds_bucket,' I also observe metrics like 'kubelet_runtime_operations_duration_seconds_sum' and 'kubelet_runtime_operations_duration_seconds_total' in Grafana. Any guidance or insights on resolving this would be greatly appreciated." enter image description here

I tried below with no success

filter/metrics:
        error_mode: ignore
        metrics:
          include:
            match_type: strict
            metric_names:
              - otelcol_receiver_accepted_metric_points_total
              - jvm_memory_used_bytes
              - jvm_threads_states_threads
              - container_cpu_usage_seconds_total
              - kubelet_runtime_operations_duration_seconds 

 filter/metrics:
        error_mode: ignore
        metrics:
          include:
            match_type: regexp
            metric_names:
              - otelcol.*
              - jvm_memory_used_bytes
              - jvm_threads_states_threads
              - container_cpu_usage_seconds_total
              - kubelet_runtime_operations_duration_seconds_buck.*
              - volume_operation_total_seconds_bucket.* 
0

There are 0 answers