So I have a cluster of 6 machines. These machines have 2 IP address (a public and a private). The private IPs are using a high speed network (1 GBps) and the public are using a slower network (120 MBps).
While configuring my kafka/zookeeper cluster, I had set the private IP as the brokers's ip. Now my configuration is fine as I can see output saying that all my brokers are connected on the correct IP
INFO Connecting to zookeeper on 10.11.10.2:2181,10.11.10.3:2181,10.11.10.4:2181,10.11.10.5:2181,10.11.10.6:2181,10.11.10.7:2181 (kafka.server.KafkaServer)
Now when I was testing the producer's performance, I could get a throughput more than 120 MBps.
bin/kafka-producer-perf-test.sh --topic highthr2 --num-records 10000 --record-size 100000 --throughput 10000 --producer-props acks=-1 bootstrap.servers=10.11.10.2:9092,10.11.10.3:9092,10.11.10.4:9092,10.11.10.5:9092,10.11.10.6:9092,10.11.10.7:9092
When I used nload, to check how much data is going through each interface, I found that data was coming in and going out through the public IP interface (slower network interface). I rechecked the zookeeper and kafka logs, but they both say that my brokers are on the private IP.
Any ideas on what is the issue or what do I need to change?