Spring Kafka: Credential Rotation

Question

We are using Aiven as a Kafka as a Service provider, and they support credential rotation. E.g. every 24 hours.

So far, we have yet to find a way to create new consumers / producers in Spring Kafka when we need to rotate. Basically, a TopicAuthorizationException is thrown on consumer.poll.

What we need to do, is basically create a new consumer with a update truststore and keystore path. Keystore and truststore is loaded from kubernetes secrets.

Anyone know how we can make this work?

Answer 1

The listener container will stop after any AuthorizationException.

You can add an ApplicationListener or @EventListener method to receive a ListenerContainerStoppedEvent.

You can either wait for all child containers to stop, or immediately stop the concurrent container; update the container's containerProperties.kafkaConsumerProperties with the new store locations and restart the container.

I don't currently see a good solution on the producer side; you would need to replace the producer factory bean or the KafkaTemplate(s) with new overrides for the store locations.

I'll have opened an issue to make it easier there.