how to change __consumer_offsets cleanup plicy to delete from compact?

Question

I have kafka clusters with 3 nodes,in kafka log directory have __consumer_offsets-XX,lots of log files as 100MB used by disk place over 90%, excutes command:

./bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-name __consumer_offsets --entity-type topics

Configs for topics:__consumer_offsets are:

segment.bytes=104857600,cleanup.policy=compact,compression.type=producer

This topic cleanup policy is compact,how to change to delete?

Answer 1

This topic is system topic for kafka. Two clean up policies are applied to it: compaction and delete(offsets.retention.minutes).

If you want to decrease size of that topic:

1. change segment.bytes to smaller amount (param offsets.topic.segment.bytes)
2. decrease retention policy for unused consumer groups (param offsets.retention.minutes)
3. try to decrease amount of partitions for this topic (param offsets.topic.num.partitions)