How does Thanos pull metrics from PushGateway?

87 views Asked by At

I have a metric: my_metric{date="", checker="", host=""}

For some business reason, the date label may be updated everyday. That means, suppose I have two metrics on 2023-10-31, like this:

my_metric{date="2023-10-31", checker="fileChecker", host="server_01"}    1
my_metric{date="2023-10-31", checker="dirChecker", host="server_02"}     2

Then the second day, my fileChecker.sh cronjob checks the status of some target files, and my metrics look like this:

my_metric{date="2023-10-31", checker="fileChecker", host="server_01"}    1
my_metric{date="2023-10-31", checker="dirChecker", host="server_02"}     2
my_metric{date="2023-11-01", checker="fileChecker", host="server_01"}    0

Now I use PushGateway to collect these metrics from my server, and let Thanos to pull them from the PushGateway. And my intention is to let Thanos be a remote storage.

To do this, I set PushGateway's grouping key as {checker, host}. I know that because the same checker from the same host will form the same grouping key, so PushGateway overwrites my metrics.

I suppose these metrics inside PushGateway should be like this:

my_metric{date="2023-10-31", checker="dirChecker", host="server_02"}     2
my_metric{date="2023-11-01", checker="fileChecker", host="server_01"}    0

But if I config Thanos to pull metrics from PushGateway every 1 minute or anyway less than 1 day. The metric

my_metric{date="2023-10-31", checker="fileChecker", host="server_01"} 1

should be stored inside Thanos. So even if it is overwritten by the same grouping key at the second day 2023-11-01, the overwritten should only happen inside PushGateway but not inside Thanos. However, when I query this metric on Grafana, and set data source as Thanos, I find it is not return this metric.

Does anyone know why this happens? I was thinking it must be the problem with same grouping key but if i understand it right, this should not happen in a remote storage Thanos.

0

There are 0 answers