SSL confluent-kafka-dotnet librdkafka SSL

Question

Im having an issue getting the confluent-kafka-dotnet library working with SSL. Everything works fine withouth SSL and i can get SSL working by using kafkas own scritps as below.

> .\kafka-console-producer.bat --broker-list domain.net:9094 --topic
busit-test --producer.config client-ssl.properties

client-ssl.properties content:

security.protocol=SSL
ssl.truststore.location=C:/Certificates/store.jks
ssl.truststore.password=mysecret

i got the ca in a file cert.crt and thats how i got the store.jks by using:

keytool -importcert -keystore store.jks -alias issuing -file cert.crt

To use confluent-kafka-dotnet i need to configure it according to the documentation ( https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka ).

metadata.broker.list=at_least_one_of_the_brokers
security.protocol=ssl

# CA certificate file for verifying the broker's certificate.
ssl.ca.location=ca-cert

# Client's certificate
ssl.certificate.location=client_?????_client.pem

# Client's key
ssl.key.location=client_?????_client.key

# Key password, if any.
ssl.key.password=abcdefgh

i transform the cert.crt file to a .pem by using:

openssl x509 -inform DER -in cert.crt -out cert.pem -text

That gives me a valid pem that i can read with a texteditor. I then point at that pem file from the ssl.ca.location The brokers are configured to not validate clients but when i start my client producer i get the following error log:

Connected to ipv4#xx.xxx.xx.xx:9094
failed: err: Local: SSL error: (errno: No error)
Broker changed state CONNECT -> DOWN

I have tried everything i can think of but cant get it to work. since it works with the scripts i can download from kafka im sure there is nothing wrong with the brokers.

Any help would be appriciated.

Answer 1

Using SSL Certificates to securely connect to Kafka using Dotnet Console App

Nuget packages to install (Package Manager Console)

Install-Package Confluent.Kafka
Install-Package Microsoft.Extensions.Configuration.Json

appsettings.json

{
  "ApplicationName": "DotNetConsoleClientForStrimziKafka",
  "KafkaClientConfig": {
    "bootstrap.servers": "<PublicIP_of_LoadBalancer>:9094",
    "security.protocol": "SSL",
    "ssl.ca.location": "C:\\kafkacerts\\ca.crt",
    "ssl.certificate.location": "C:\\kafkacerts\\user.crt",
    "ssl.key.location": "C:\\kafkacerts\\user.key"
  }
}

Program.cs

appsettingsConfig = new ConfigurationBuilder()
  .SetBasePath(System.AppContext.BaseDirectory)
  .AddJsonFile(DefaultSettingsJsonFile, false, true)
  .Build();


var configProperties = appsettingsConfig.GetSection(configSectionName).GetChildren().ToDictionary(x => x.Key, x => x.Value);

var producerConfig = new ProducerConfig(configProperties);

using var producer = new ProducerBuilder<string, string>(producerConfig).Build();

You need to download and put decoded certificates in folder (C:\kafkacerts in my case).

See details: Using SSL Certificates For Kafka Dotnet Client