I have a metric: my_metric{date="", checker="", host=""}
For some business reason, the date
label may be updated everyday. That means, suppose I have two metrics on 2023-10-31
, like this:
my_metric{date="2023-10-31", checker="fileChecker", host="server_01"} 1
my_metric{date="2023-10-31", checker="dirChecker", host="server_02"} 2
Then the second day, my fileChecker.sh
cronjob checks the status of some target files, and my metrics look like this:
my_metric{date="2023-10-31", checker="fileChecker", host="server_01"} 1
my_metric{date="2023-10-31", checker="dirChecker", host="server_02"} 2
my_metric{date="2023-11-01", checker="fileChecker", host="server_01"} 0
Now I use PushGateway
to collect these metrics from my server, and let Thanos
to pull them from the PushGateway
. And my intention is to let Thanos
be a remote storage.
To do this, I set PushGateway's grouping key
as {checker, host}
. I know that because the same checker
from the same host
will form the same grouping key
, so PushGateway
overwrites my metrics.
I suppose these metrics inside PushGateway should be like this:
my_metric{date="2023-10-31", checker="dirChecker", host="server_02"} 2
my_metric{date="2023-11-01", checker="fileChecker", host="server_01"} 0
But if I config Thanos to pull metrics from PushGateway every 1 minute or anyway less than 1 day. The metric
my_metric{date="2023-10-31", checker="fileChecker", host="server_01"} 1
should be stored inside Thanos
. So even if it is overwritten by the same grouping key at the second day 2023-11-01, the overwritten should only happen inside PushGateway
but not inside Thanos
. However, when I query this metric on Grafana
, and set data source as Thanos
, I find it is not return this metric.
Does anyone know why this happens? I was thinking it must be the problem with same grouping key but if i understand it right, this should not happen in a remote storage Thanos.