I have multiple targets in prometheus which generate multiple metrics. I need to verify the values generated by a certain metric on multiple instances and trigger an alert incase the values are not equal to each other.

metric_name: treds_load_peer_db_doc_cnt

values log:

treds_load_peer_db_doc_cnt{instance="com.peer0",ip="",job="prod"} 2136589 treds_load_peer_db_doc_cnt{instance="com.peer1",ip="",job="prod"} 2136590 treds_load_peer_db_doc_cnt{instance="com.peer2",ip="",job="prod"} 2136590

here's the query i'm using currently: treds_load_peer_db_doc_cnt{instance="com.peer0"} != ignoring(instance,ip) treds_load_peer_db_doc_cnt{instance="com.peer1"}

which works out but messes up all the labels. Is there a way to check metric in all targets at once & alert in case of miss-match?

1 Answers

brian-brazil On Best Solutions

I'd do something like:

max without(instance,ip)(treds_load_peer_db_doc_cnt) != min without(instance,ip)(treds_load_peer_db_doc_cnt)

which will generate an alert if they aren't all the same.