promQL opetrate two metric with same labels, but different label-val combiantions,

Question

I have to metric one metric is total count, another is error count, they have same labels, but different label-val combination, total count metric's label-vals combinations mush constains error count metric's label-val combination,such as:

total count metric:

method_totalCall{host="h1", dst="a1", src="a2"}@t1 = 24
method_totalCall{host="h1", dst="b1", src="b2"}@t1 = 30
method_totalCall{host="h1", dst="c1", src="c2"}@t1 = 3
method_totalCall{host="h1", dst="d1", src="d2"}@t1 = 6
method_totalCall{host="h1", dst="e1", src="e2"}@t1 = 21

this metric means method call count in $host from $src to $dest

error count metric:

method_totalCall_ERROR{host="h1", dst="c1", src="c2"}@t1 = 1
method_totalCall_ERROR{host="h1", dst="d1", src="d2"}@t1 = 2
method_totalCall_ERROR{host="h1", dst="e1", src="e2"}@t1 = 3

this metric means method call error count in $host from $src to $dest

i want to get a metric which means method call success count in $host from $src to $dest so i write a promQL:

method_totalCall{host='h1'} - method_totalCall_ERROR{host='h1'}

but if i use promQL :

method_totalCall{host='h1'} - method_totalCall_ERROR{host='h1'}

I will get a new temporary metric :

methed_succ_count{host="h1", dst="c1", src="c2"}@t1 = 2 
methed_succ_count{host="h1", dst="d1", src="d2"}@t1 = 4
methed_succ_count{host="h1", dst="e1", src="e2"}@t1 = 18


(come from:method_totalCall{host="h1", dst="c1", src="c2"}@t1 - method_totalCall_ERROR{host="h1", dst="c1", src="c2"}@t1)
(come from:method_totalCall{host="h1", dst="d1", src="d2"}@t1 - method_totalCall_ERROR{host="h1", dst="d1", src="d2"}@t1)
(come from:method_totalCall{host="h1", dst="e1", src="e2"}@t1 - method_totalCall_ERROR{host="h1", dst="e1", src="e2"}@t1)

but for label

{host="h1", dst="a1", src="a2"}
{host="h1", dst="b1", src="b2"}

value missed in new metric. which means i doesn't get the sucess count in host h1 from from specific $src(a1, b1) to specific $dest(a2,b2).how should I modify promQL? i want get val like this:

methed_succ_count{host="h1", dst="a1", src="a2"}@t1 = 24
methed_succ_count{host="h1", dst="b1", src="b2"}@t1 = 30
methed_succ_count{host="h1", dst="c1", src="c2"}@t1 = 2 
methed_succ_count{host="h1", dst="d1", src="d2"}@t1 = 4
methed_succ_count{host="h1", dst="e1", src="e2"}@t1 = 18

which means when no error occur on host h1 from some specific $src(a1, b1) to specific $dst (a2,b2)， error count is zero from (a1,b1) to (a2, b2), success count from (a1, b1) to (a2, b2) is total count.

the tsdb i used is victoriametrics.

Answer 1

Essentially what you are trying to do, is to create a left join. PromQL doesn't have built-in operator for this, but it can be emulated with or.

In your particular case, use

method_totalCall{host='h1'} - method_totalCall_ERROR{host='h1'}
 or method_totalCall{host='h1'}

For more details on the matter check this post by Brian Brasil.

---

A sidenote, regarding metric naming: you metric names don't conform to the best practices.

Additionally, consider (if possible) changing your instrumentation, so that you actually expose metric with the single name, but additional label result that would contain values Success or Failure. This way you have values split by results, and if you need aggregation for total call, you can use sum without (result) (method_calls_total).