Removing stale reported metric data from opentelemetry js prometheus exporter

125 views Asked by At

I am attempting to set-up the Grafana Geomap to display the location of moving units. I am using the OpenTelemetry JS npm package with the Prometheus Exporter. The problem I am running into is removing 'stale' metrics from being continuously reported. I can not remove callback functions that have been pulled by the metrics endpoint.

I created a metrics controller class that creates the metrics from the meter, creates an object to hold the callback functions, and sets a timeout to cleanup old callbacks.

export class MetricsController {
  constructor(meter) {
    this.callbacks = {};
    this.reportingMetrics = ["metric1", "metric2", "location", "latitude", "longitude"];
    this.metrics = this.createMetrics(meter);

    setTimeout(this.cleanupMetrics.bind(this), 10000);
  }
  createMetrics(meter) {
    const metrics = {};
    for (const metric of this.reportingMetrics) {
      metrics[metric] = meter.createObservableGauge(
        "unit_" + metric,
        {
          description: "unit " + metric
        }
      );
    }
    return metrics;

  }
  recordMetrics(incomingMetrics) {
    if (!this.callbacks[incomingMetrics.unitId])
      this.callbacks[incomingMetrics.unitId] = {};

    this.callbacks[incomingMetrics.unitId].timestamp = new Date();

    for (const metric of this.reportingMetrics) {
      if (this.callbacks[incomingMetrics.unitId][metric]) {
        this.metrics[metric].removeCallback(
          this.callbacks[incomingMetrics.unitId][metric]
        );
        delete this.callbacks[incomingMetrics.unitId][metric];
      }
      if (metric == "location") {
        this.callbacks[incomingMetrics.unitId][metric] = (observableResult) => {
          observableResult.observe(1, {
            key: incomingMetrics.unitId,
            latitude: incomingMetrics.latitude,
            longitude: incomingMetrics.longitude,
            name: incomingMetrics.unitId
          });
        };
      } else {
        this.callbacks[incomingMetrics.unitId][metric] = (observableResult) => {
          observableResult.observe(incomingMetrics[metric], {
            unit: incomingMetrics.unitId
          });
        };
      }
      this.metrics[metric].addCallback(
        this.callbacks[incomingMetrics.unitId][metric]
      );
    }
  }
  cleanupMetrics() {
    for (const unit of Object.keys(this.callbacks)) {
      if (
        Date.now() - this.callbacks[unit].timestamp <=
        process.env.METRICS_TIMEOUT_MINUTES * 60 * 1000
      ) {
        for (const metric of this.reportingMetrics) {
          this.metrics[metric].removeCallback(this.callbacks[unit][metric]);
        }
        delete this.callbacks[unit];
      }
    }
    setTimeout(this.cleanupMetrics.bind(this), 10000);
  }
}

This works almost the way I expected, however there is one issue. If a metric is reported to the metrics endpoint, I can't remove the callback function anymore. It will continue reporting it forever. Is there a way to clean up this stale data? It's very important for the location because every new lat/lon creates a new metric since they are being attached as labels. I'm attaching the lat/lon as labels so I can display them on the geomap, but maybe there is a better way to do that?

This also means I won't know when units are off since their metrics will be keep being reported by the exporter. The only method I could find to purge data was this removeCallback function but I might have missed something, I am very new to prometheus/metrics so I also might just be thinking about this incorrectly. I'd love it if the exporter just purged the metrics after each scrape, then I wouldn't even need this controller.

0

There are 0 answers