As mentioned in the title, the segment file is not deleted for a long time, which makes the partition size very large(larger than 100G).
In my opinion, __consumer_offsets cleanup policy is compact, and those messages inactive should be deleted after the expiration time. The value default is 1 day, which is hard code.
val OffsetsRetentionMinutes: Int = 24 * 60 //KafkaConfig.scala
offsetsRetentionMs = config.offsetsRetentionMinutes * 60 * 1000L //GroupCoordinator.scala
From the image below, however, I consume the message from __consume_offsets, the expire time is far beyond 1 day, it is still not deleted.
Enviroment:
kafka 0.9.0.1
jdk 1.7.0_80
Check
log.segment.bytes
. By default it is1G
and segment is closed after it reaches to that size. Retention policies are not applied to segments which are not closed.