I am attempting to set-up the Grafana Geomap to display the location of moving units. I am using the OpenTelemetry JS npm package with the Prometheus Exporter. The problem I am running into is removing 'stale' metrics from being continuously reported. I can not remove callback functions that have been pulled by the metrics endpoint.
I created a metrics controller class that creates the metrics from the meter, creates an object to hold the callback functions, and sets a timeout to cleanup old callbacks.
export class MetricsController {
constructor(meter) {
this.callbacks = {};
this.reportingMetrics = ["metric1", "metric2", "location", "latitude", "longitude"];
this.metrics = this.createMetrics(meter);
setTimeout(this.cleanupMetrics.bind(this), 10000);
}
createMetrics(meter) {
const metrics = {};
for (const metric of this.reportingMetrics) {
metrics[metric] = meter.createObservableGauge(
"unit_" + metric,
{
description: "unit " + metric
}
);
}
return metrics;
}
recordMetrics(incomingMetrics) {
if (!this.callbacks[incomingMetrics.unitId])
this.callbacks[incomingMetrics.unitId] = {};
this.callbacks[incomingMetrics.unitId].timestamp = new Date();
for (const metric of this.reportingMetrics) {
if (this.callbacks[incomingMetrics.unitId][metric]) {
this.metrics[metric].removeCallback(
this.callbacks[incomingMetrics.unitId][metric]
);
delete this.callbacks[incomingMetrics.unitId][metric];
}
if (metric == "location") {
this.callbacks[incomingMetrics.unitId][metric] = (observableResult) => {
observableResult.observe(1, {
key: incomingMetrics.unitId,
latitude: incomingMetrics.latitude,
longitude: incomingMetrics.longitude,
name: incomingMetrics.unitId
});
};
} else {
this.callbacks[incomingMetrics.unitId][metric] = (observableResult) => {
observableResult.observe(incomingMetrics[metric], {
unit: incomingMetrics.unitId
});
};
}
this.metrics[metric].addCallback(
this.callbacks[incomingMetrics.unitId][metric]
);
}
}
cleanupMetrics() {
for (const unit of Object.keys(this.callbacks)) {
if (
Date.now() - this.callbacks[unit].timestamp <=
process.env.METRICS_TIMEOUT_MINUTES * 60 * 1000
) {
for (const metric of this.reportingMetrics) {
this.metrics[metric].removeCallback(this.callbacks[unit][metric]);
}
delete this.callbacks[unit];
}
}
setTimeout(this.cleanupMetrics.bind(this), 10000);
}
}
This works almost the way I expected, however there is one issue. If a metric is reported to the metrics endpoint, I can't remove the callback function anymore. It will continue reporting it forever. Is there a way to clean up this stale data? It's very important for the location because every new lat/lon creates a new metric since they are being attached as labels. I'm attaching the lat/lon as labels so I can display them on the geomap, but maybe there is a better way to do that?
This also means I won't know when units are off since their metrics will be keep being reported by the exporter. The only method I could find to purge data was this removeCallback function but I might have missed something, I am very new to prometheus/metrics so I also might just be thinking about this incorrectly. I'd love it if the exporter just purged the metrics after each scrape, then I wouldn't even need this controller.