MANAGE LIFECYCLE OF VAULT SECRET WITH CUSTOM ["org.apache.kafka.common.config.provider"]

Question

My question concerns the "org.apache.kafka.common.config.provider" interface. As seen in the documentation: https://kafka.apache.org/35/javadoc/org/apache/kafka/common/config/provider/ConfigProvider.html

public interface ConfigProvider extends Configurable, Closeable

This ConfigProvider interface provides the following methods:

ConfigData get(String path)

Retrieves the data at the given path.

ConfigData get(String path, Set<String> keys)

Retrieves the data with the given keys at the given path.

void configure(Map<String,?> configs)

Configures this class with the given key-value pairs.

I would like to create a Vault config provider for dynamic database secrets.

For the void configure(Map<String,?> configs) method, I will configure (by adding the vault connection parameters) the class that allows me to retrieve the secrets.

For the ConfigData get(String path) and ConfigData get(String path, Set keys) methods, it will be about defining how to retrieve the secrets in Vault.

I have questions regarding the lifecycle of secrets. Assuming I develop a method capable of telling me when a Vault secret is modified or renewed, how can I integrate it to trigger my custom ConfigProvider so that it can fetch the correct values and update the configuration of the Kafka Connect connector using this provider?

Do you have any code examples I can use as inspiration?

Answer 1

I don’t know whether I asked my question correctly or I was misunderstood, but: I’m not looking for an example of a GitHub project to develop a custom provider; I’m looking for a way to manage the secret update on an ongoing basis.

I finally found an alternative solution. We can update the values. The solution is in the use of the

public ConfigData(Map<String,String> data, Long ttl)

…in the return of configurations in the get method.

This will force the provider to fetch each "ttl ms" with the new values.