i have 3 nodes, 2 of them in the same lan and the 3rd are remote(different country). on the 2 nodes: the 1st node runs a docker compose with some services that exports metrics, and another compose with prometheus and thanos sidecar which gather the metrics and export them to minio. on the 2nd node there is a minio instance (runs as a container) and a python script that backup the data to aws s3 bucket when there is internet connection. in the 3rd nodes i have a docker compose with thanos-query and thanos-store which configure to receive the data from aws s3 bucket. i can see the data in aws in the same format as in the minio bucket. it seems that the store is not receiving the data correctly or at all. i can see the store in thanos-query but no data. this is my prometheus-compose.yaml
services:
prometheus:
image: prom/prometheus:v2.51.0
container_name: prometheus
restart: always
user: root
ports:
- "9090:9090"
volumes:
- ./data/prometheus:/data
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./alerts:/etc/prometheus/rules/
command:
- --config.file=/etc/prometheus/prometheus.yml
- --web.enable-admin-api
- --storage.tsdb.path=/data
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention.time=2h'
- '--web.enable-lifecycle'
- '--web.listen-address=:9090'
- '--storage.tsdb.min-block-duration=5m'
- '--storage.tsdb.max-block-duration=5m'
thanos-sidecar:
image: quay.io/thanos/thanos:v0.34.1
container_name: thanos-sidecar
restart: always
user: 0:1001
environment:
- MINIO_BUCKET_NAME=${MINIO_BUCKET_NAME}
- MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY}
- MINIO_SECRET_KEY=${MINIO_SECRET_KEY}
- MINIO_ENDPOINT=${MINIO_ENDPOINT}
command:
- "sidecar"
- "--log.level=debug"
- "--tsdb.path=/data"
- "--prometheus.url=http://prometheus:9090"
- "--reloader.config-file=/etc/config/prometheus.yml"
- |
--objstore.config=type: S3
config:
bucket: ${MINIO_BUCKET_NAME}
access_key: ${MINIO_ACCESS_KEY}
secret_key: ${MINIO_SECRET_KEY}
endpoint: ${MINIO_ENDPOINT}
insecure: true
volumes:
- ./data/prometheus:/data
- ./prometheus.yml:/etc/config/prometheus.yml:ro
- ./thanos/sidecar:/etc/thanos
expose:
- 10902
- 10901
thanos-query.yaml:
services:
thanos-store:
image: quay.io/thanos/thanos:v0.34.1
container_name: thanos-store
restart: always
user: 0:1001
volumes:
- ./thanos/store:/etc/thanos
- ./bucket.yml:/etc/prometheus/bucket.yml
command:
- "store"
- "--log.level=debug"
- "--data-dir=/data"
- "--log.format=logfmt"
- "--index-cache-size=250MB"
- "--chunk-pool-size=1GB"
- "--store.grpc.series-max-concurrency=20"
- "--sync-block-duration=3m"
- "--block-sync-concurrency=20"
- "--objstore.config-file=/etc/prometheus/bucket.yml"
expose:
- 10902
- 10901
ports:
- '10912:10902'
thanos-query:
image: quay.io/thanos/thanos:v0.34.1
container_name: thanos-query
restart: always
command:
- "query"
- "--log.level=debug"
- "--log.format=logfmt"
- "--store=thanos-store:10901"
- "--store.sd-interval=5m"
- "--query.replica-label=monitor"
expose:
- 10902
- 10901
ports:
- "10902:10902"
my bucket.yaml:
type: S3
config:
bucket: "<my-bucket>"
endpoint: "s3.<my-region>.amazonaws.com"
region: "<my-region>"
access_key: "<my_key>"
secret_key: "<my_secret>"
tested and double checked that the permission to the bucket is not the problem. i suspect the problem might be that the python script is not copying the data in the same manner as thanos-sidecar, though the format looks the same. when i configured thanos-query on the same host as prometheus i get the data from minio just fine (2h retention time so i know that im getting from minio and from thanos-sidecar). end to end everything looks configured ok exept that i cant read metrics from aws s3. there are no error in logs of thanos-store and query and i can see the store in thanos-query(in the 3rd node) is this setup even possible? (the use of standalone thanos-query with thanos-store just for read) thank you