What if we don't explain any external_labels in Prometheus for Thanos?

3.7k views Asked by At

If we don't explain any external_labels in Prometheus configuration file for Thanos, what will happen?

For example, I have two Prometheuses and they are scraping the same endpoint, so their external_labels is like this:

prometheus1:

global:
  external_labels:
    replica: 1

prometheus2:

global:
  external_labels:
    replica: 2

At this point, we know that two kind of metrics will be saved in the object storage, metrics with replica=1 label and replica=2 label. If I delete the external_labels from configuration file, how many metrics will be saved in the object storage?

1

There are 1 answers

1
Hedeesa On

I found the answer.
based on this doc,

Every Prometheus instance must have a globally unique set of identifying labels.

And if you don't declare one for your Prometheus, TSDB blocks wont be uploaded to your object storage and you gonna receive this error from sidecar:

level=warn ts=2020-10-03T17:00:14.826489487Z caller=sidecar.go:274 err="upload 01EKQPFYHW6R8TB48N07NSCH8P: empty external labels are not allowed for Thanos block." uploaded=0