How to configure confluentinc/cp-kafka with SASL_PLAINTEXT authentication in docker-compose?

258 views Asked by At

I have working configuration for confluentinc/cp-kafka that works fine in docker-compose as following:

User
  kafka-kraft:
    image: confluentinc/cp-kafka:7.5.1
    container_name: kafka-kraft
    hostname: kafka-kraft
    ports:
      - "9101:9101"
      - "9092:9092"
    environment:
      - KAFKA_NODE_ID=1
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-kraft:29092,PLAINTEXT_HOST://localhost:9092
      - KAFKA_JMX_PORT=9101
      - KAFKA_JMX_HOSTNAME=localhost
      - KAFKA_PROCESS_ROLES=broker,controller
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka-kraft:29093
      - KAFKA_LISTENERS=PLAINTEXT://kafka-kraft:29092,CONTROLLER://kafka-kraft:29093,PLAINTEXT_HOST://0.0.0.0:9092
      - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
      - CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk

Now I try to configure it so it will work with SASL_PLAINTEXT authentication, tried to change it to the following configuration:

  kafka-kraft:
    image: confluentinc/cp-kafka:7.5.1
    container_name: kafka-kraft
    hostname: kafka-kraft
    ports:
      - "9101:9101"
      - "9092:9092"
    environment:
      - KAFKA_NODE_ID=1
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT_HOST:SASL_PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=SASL_PLAINTEXT://kafka-kraft:29092,PLAINTEXT_HOST://localhost:9092
      - KAFKA_JMX_PORT=9101
      - KAFKA_JMX_HOSTNAME=localhost
      - KAFKA_PROCESS_ROLES=broker,controller
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka-kraft:29093
      - KAFKA_LISTENERS=SASL_PLAINTEXT://kafka-kraft:29092,CONTROLLER://kafka-kraft:29093,PLAINTEXT_HOST://0.0.0.0:9092
      - KAFKA_INTER_BROKER_LISTENER_NAME=SASL_PLAINTEXT
      - KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
      - CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk
      - KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN
      - KAFKA_SASL_ENABLED_MECHANISMS=PLAIN
      - KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
    volumes:
      - ./kafka_server_jaas.conf:/etc/kafka/kafka_server_jaas.conf

kafka_server_jaas.conf:

KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-secret"
    user_admin="admin-secret"
    user_kafka="kafka-secret";
};

Now when I try to run it, I get the following error:

===> User
2023-11-29T10:52:36.043745169Z uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
2023-11-29T10:52:36.043961086Z ===> Configuring ...
2023-11-29T10:52:36.047186169Z Running in KRaft mode...
2023-11-29T10:52:36.625520503Z CLUSTER_ID is required.
2023-11-29T10:52:36.625564128Z Command [/usr/local/bin/dub ensure CLUSTER_ID] FAILED !

Honestly, I first time try to configure Kafka for developing environment and have no clue what I need to fix to cause it to work, and why adding security have an impact on CLUSTER_ID at all?

0

There are 0 answers