Springboot gateway Prometheus collecting huge data

Question

I am using springboot as microservice.

I have around 20 microservices in my k8s cluster.

I am using promethues to collect data to show in grafana.

In that application there are some url which uses Path variable like as follow

1. /v1/contacts/{id}
2. /v1/users/{id}

There are few more urls, If I consider all such URLs in all microservices then that could be around 60 to 70 URLs which uses path variable.

Problem :

Now whenever any url is getting requested i.e like

1. /v1/contacts/10
2. /v1/contacts/111
3. /v1/contacts/51
4. /v1/users/91

so on...

Then promethus is collecting metrics for all such url. After some time it has huge metrics, and at the end my response time is increased for collecting data from prometheus.

So basically I want to clear prometheus logs after some interval from my springboot application.

I am not sure whether its possible or not.

Can someone please help me to solve this ?

Thanks

Answer 1

Prometheus has several flags that configure local storage.

--storage.tsdb.path: Where Prometheus writes its database. Defaults to data/.
--storage.tsdb.retention.time: When to remove old data. Defaults to 15d. Overrides storage.tsdb.retention if this flag is set to anything other than default.
--storage.tsdb.retention.size: [EXPERIMENTAL] The maximum number of bytes of storage blocks to retain. The oldest data will be removed first. Defaults to 0 or disabled. This flag is experimental and may change in future releases. Units supported: B, KB, MB, GB, TB, PB, EB. Ex: "512MB"

Prometheus storage link

Steps:

1. Spring Boot application exposure monitoring indicators, add the following dependencies.
   <artifactId>spring-boot-starter-actuator</artifactId>
   or
   <groupId>io.prometheus</groupId>
artifactId>simpleclient_spring_boot</artifactId>

2. Prometheus collects Spring Boot indicator data. First, get the Docker image of Prometheus: docker pull prom/prometheus

   Then, write the configuration file prometheus.yml:

   global:
     scrape_interval: 10s
     scrape_timeout: 10s
     evaluation_interval: 10m
   
   scrape_configs:
     - job_name: prometheus
       scrape_interval: 5s
       scrape_timeout: 5s
       metrics_path: /metrics
       scheme: http
   

   You can add more values to it. Sample here

3. Next, start Prometheus:

   docker run -d -p 9090:9090 \
   -u root \
   -v /opt/prometheus/tsdb:/etc/prometheus/tsdb \
   -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
   --privileged=true prom/prometheus \
   --storage.tsdb.path=/etc/prometheus/tsdb \
   --storage.tsdb.retention.time=7d \
   --storage.tsdb.retention.size = 300MB \
   --config.file=/etc/prometheus/prometheus.yml
   

Alternative way, you can use this link Monitoring Spring Boot projects with Prometheus
But make sure when you run the Prometheus server with Docker Compose you have to update the commands section with size or time properties.