How can SASL/PLAIN credentials be provided when connecting from pipelinedb to kafka?

Question

We want to ingest data from Kafka through PipelineDB. Our Kafka setup requires SASL/PLAIN authentication. How can I configure the SASL user and password with the pipelinedb / pipeline_kafka PostgreSQL extension?

Answer 1

We were able to do it like this:

Apply this patch to pipeline_kafka (the related issue was closed, but the patch was never merged).

Then add this configuration to the postgresql.conf:

shared_preload_libraries = 'pipelinedb,pipeline_kafka'
pipeline_kafka.broker_version = '1.0.0'
pipeline_kafka.consumer_config = 'security.protocol=SASL_PLAINTEXT,sasl.mechanisms=PLAIN,sasl.username=USER,sasl.password=PASSWORD'

Probably there's a better answer, but this worked.