kafka __consumer_offsets segment file not deleted after expiration time

Question

As mentioned in the title, the segment file is not deleted for a long time, which makes the partition size very large(larger than 100G).

In my opinion, __consumer_offsets cleanup policy is compact, and those messages inactive should be deleted after the expiration time. The value default is 1 day, which is hard code.

  val OffsetsRetentionMinutes: Int = 24 * 60  //KafkaConfig.scala

  offsetsRetentionMs = config.offsetsRetentionMinutes * 60 * 1000L //GroupCoordinator.scala

From the image below, however, I consume the message from __consume_offsets, the expire time is far beyond 1 day, it is still not deleted.

Enviroment:

• kafka 0.9.0.1

• jdk 1.7.0_80

Answer 1

Check log.segment.bytes. By default it is 1G and segment is closed after it reaches to that size. Retention policies are not applied to segments which are not closed.