I have the following configuration for my OpenTelemetry (otel) filter processor. Within the receiver configuration, kubelet is scraping node metrics, and the filter is intended to selectively include or exclude metrics based on the filter configuration.
Here's the scrape configuration job in the receiver:
job_name: kubelet
scrape_interval: 1m
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
api_server: null
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
action: replace
target_label: __metrics_path__
regex: (ip-.*)
replacement: /api/v1/nodes/$${1}/proxy/metrics
In the filter configuration:
filter/metrics:
error_mode: ignore
metrics:
include:
match_type: regexp
metric_names:
- otelcol_receiver_accepted_metric_points_total$
- jvm_memory_used_bytes$
- jvm_threads_states_threads$
- container_cpu_usage_seconds_total$
- kubelet_runtime_operations_duration_seconds_bucket$
However, I'm encountering an issue. While I expect to see the metric name 'kubelet_runtime_operations_duration_seconds_bucket,' I also observe metrics like 'kubelet_runtime_operations_duration_seconds_sum' and 'kubelet_runtime_operations_duration_seconds_total' in Grafana. Any guidance or insights on resolving this would be greatly appreciated."
I tried below with no success
filter/metrics:
error_mode: ignore
metrics:
include:
match_type: strict
metric_names:
- otelcol_receiver_accepted_metric_points_total
- jvm_memory_used_bytes
- jvm_threads_states_threads
- container_cpu_usage_seconds_total
- kubelet_runtime_operations_duration_seconds
filter/metrics:
error_mode: ignore
metrics:
include:
match_type: regexp
metric_names:
- otelcol.*
- jvm_memory_used_bytes
- jvm_threads_states_threads
- container_cpu_usage_seconds_total
- kubelet_runtime_operations_duration_seconds_buck.*
- volume_operation_total_seconds_bucket.*