I have a metric: my_metric{date="", checker="", host=""}
For some business reason, the date label may be updated everyday. That means, suppose I have two metrics on 2023-10-31, like this:
my_metric{date="2023-10-31", checker="fileChecker", host="server_01"} 1
my_metric{date="2023-10-31", checker="dirChecker", host="server_02"} 2
Then the second day, my fileChecker.sh cronjob checks the status of some target files, and my metrics look like this:
my_metric{date="2023-10-31", checker="fileChecker", host="server_01"} 1
my_metric{date="2023-10-31", checker="dirChecker", host="server_02"} 2
my_metric{date="2023-11-01", checker="fileChecker", host="server_01"} 0
Now I use PushGateway to collect these metrics from my server, and let Thanos to pull them from the PushGateway. And my intention is to let Thanos be a remote storage.
To do this, I set PushGateway's grouping key as {checker, host}. I know that because the same checker from the same host will form the same grouping key, so PushGateway overwrites my metrics.
I suppose these metrics inside PushGateway should be like this:
my_metric{date="2023-10-31", checker="dirChecker", host="server_02"} 2
my_metric{date="2023-11-01", checker="fileChecker", host="server_01"} 0
But if I config Thanos to pull metrics from PushGateway every 1 minute or anyway less than 1 day. The metric
my_metric{date="2023-10-31", checker="fileChecker", host="server_01"} 1
should be stored inside Thanos. So even if it is overwritten by the same grouping key at the second day 2023-11-01, the overwritten should only happen inside PushGateway but not inside Thanos. However, when I query this metric on Grafana, and set data source as Thanos, I find it is not return this metric.
Does anyone know why this happens? I was thinking it must be the problem with same grouping key but if i understand it right, this should not happen in a remote storage Thanos.