I did set up 5 replicas of Thanos Receivers with 3 hours of data retention.
I verified that ULID folders created every 2 hours are being created and being deleted after 3 hours.
However, my problem is, the data in the wal directory seems keep on increasing and not being deleted after 3 hours. I was trying to find any information about it in Thanos Documentation if this is an expected behavior. We only provision 20Gi of pvc, and we cannot afford to increase it as part of company's cost effectivity goal.
Is there any information where in I can make the wal directory in Thanos Receivers data storage can be deleted as the ULID directories do?
Expectation: Data folders inside the wal directory would be deleted the same way as the ULID directories get deleted after 3 hours.
Please give me some light. Thank you.