Multiple instance with Spring Cloud Bus Kafka

Question

My question is how to manage the multi instance with Spring Cloud Stream Kafka.

Let me explain, in a Spring Cloud Stream Microservices context (eureka, configserver, kafka) I want to have 2 instances of the same microservice. When I change a configuration in my GIT Repository, the configserver (via a webhook) will push a message into the Kafka topic.

If i use the same group-id in my microservice, only one of two instances will received the notification, and reload his spring context. But I need to refresh all instances ...

So, to do that, I have configured an unique group-id : ${spring.application.name}.bus.${hostname} It's work well, but the problem is, each time I start a new instance of my service, it create a new consumer group in kafka. Now i have a lot of unused consumer group.

[![consumers for a microservice][1]][1] [1]: https://i.stack.imgur.com/6jIzx.png

Here is the Spring Cloud Stream configuration of my service :

spring:
  cloud:
    bus:
      destination: sys.spring-cloud-bus.refresh
      enabled: true
      refresh:
        enabled: true
      env:
        enabled: true
      trace:
        enabled: false
    stream:
      bindings:
        # Override spring cloud bus configuration with a specific binder named "bus"
        springCloudBusInput:
          binder: bus
          destination: sys.spring-cloud-bus.refresh
          content-type: application/json
          group: ${spring.application.name}.bus.${hostname}
        springCloudBusOutput:
          binder: bus
          destination: sys.spring-cloud-bus.refresh
          content-type: application/json
          group: ${spring.application.name}.bus.${hostname}
      binders:
        bus:
          type: kafka
          defaultCandidate: false
          environment:
            spring:
              cloud:
                stream:
                  kafka:
                    binder:
                      brokers: kafka-dev.hcuge.ch:9092
      kafka:
        streams:
          bindings:
            springCloudBusInput:
              consumer:
                startOffset: latest # Reset offset to the latest value to avoid consume configserver notifications on startup
                resetOffsets: true

How to avoid lot of consumer creation ? Should I remove old consumer group in kafka ? I think my solution is not the best way to do it, so if you have a better option, I'm interested;)

Thank you

Answer 1

If you don't provide a group, bus will use a random group anyway.

The broker will eventually remove the unused groups according to its offsets.retention.minutes property (currently 7 days by default).