Cannot connect Prometheus to Kafka broker (Get "http://host:port/metrics" : EOF) using docker compose

Question

My setup is a docker container of confluentinc/cp-kafka:7.3.2 image with the following docker-compose.yml:

version: '3'
services:
  broker-1:
    image: confluentinc/cp-kafka:7.3.2
    container_name: broker-1
    user: root
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=10.200.125.24:2181,10.200.125.11:2181,10.200.115.66:2181
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT  
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://10.200.125.24:9092,PLAINTEXT_INTERNAL://broker-1:29092
      - KAFKA_JMX_PORT=9101
      - KAFKA_JMX_REMOTE_HOSTNAME=10.200.125.24
    volumes:
      - ./kafka_data:/var/lib/kafka/data
volumes:
  kafka_data:

and a docker container of prom/prometheus image with the following docker-compose.yml:

version: '3'

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus:/etc/prometheus
    ports:
      - "9090:9090"

and the following config file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'kafka'
    static_configs:
      - targets: ['10.200.125.24:9101']

My cluster consists of 3 broker nodes and 3 zookeeper nodes and seem to be working properly. I verified that jmxremote must be working properly on 10.200.125.24:9101 by running the following jmxterm commands in the host machine: jmxterm results

I also verified that from within the prometheus container I can succssfully telnet on 10.200.125.24:9101

However, the problem is that Prometheus cannot connect to kafka metrics url (http://10.200.125.24:9101/metrics) and all I get is the following error:

Get "http://10.200.125.24:9101/metrics": EOF

prometheus screenshot

I tried to curl (http://10.200.125.24:9101/metrics) from the host machine but all i get is this:

curl --noproxy '*' http://10.200.125.24:9101/metrics
curl: (52) Empty reply from server

Any ideas on what might be wrong?

I tried to connect prometheus to kafka broker and I expected to read metrics in prometheus. Prometheus cannot connect to the kafka broker.