SSL Setup between Filebeat and Amazon MSK (Kafka)

Question

I have successfully got Filebeat exporting logs to MSK in plaintext mode. When trying to do the same over SSL I’m getting error messages from the Filebeat side

Publish failed with circuit breaker is open

and SSL Handshake errors in the MSK Cloudwatch Logs.

I don’t need two way verification so I assume I just need to pass Filebeat the ACM-PCA from the MSK console however I still receive SSL handshake problems.

Could anyone please provide a sample Filebeat config for outputs to MSK.

Answer 1

You can create a private certifcate via the ACM Console.

Once done you can export the certificate from ACM and use the cert and key in your filebeat output.

Step One

certificate manager > request a certificate > request a private certificate > select your kafka private ca

For the DNS name wildcard your MSK host name, for example

*.my-kafka-cluster.h6q2kl.c4.kafka.eu-west-2.amazonaws.com

Once done request the certificate

Step Two

Export the Private certificate under actions in certificate manager, click through and you will get the cert, cert chain and key. You will only need the cert and key.

Step Three

Add your cert and key to your instance and configure your filebeat output to use the SSL certs. (You may want to remove the password from the key) if not configure the filebeat keystore to avoid plaintext passwords

openssl rsa -in [original.key] -out [new.key]

output.kafka:
  hosts: ["*******"]

  topic: "*****"
  partition.round_robin:
    reachable_only: false

  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

  ssl.certificate: "/root/certs/kafka-broker.pem"
  ssl.key: "/root/certs/kafka.key"

You should start to see your data in your topic after you restart filebeat