Does Spring Cloud Dataflow Local Server need to connect zookeeper?

Question

I am trying to set up scdf on docker containers.

All the containers are in one network

The two kafkas are exposed by port 2181

and I am sure the kafkas are connecting to the zookeepers. (if not, they will not be started normally)

But during the creation of the simple stream

stream create --name test --definition "time | log" --deploy

I got these logs in the stdout_0.log file which stands for the log app logs.

2016-12-14 01:18:03.264  INFO 62 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-12-14 01:18:03.265  WARN 62 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn          : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_111-internal]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_111-internal]
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[zookeeper-3.4.6.jar!/:3.4.6-1569965]

in stderr_1.log, I have

Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 10000
    at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
    at org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder.ensureTopicCreated(KafkaMessageChannelBinder.java:354)
    at org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder.createKafkaConsumer(KafkaMessageChannelBinder.java:441)
    at org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder.doBindConsumer(KafkaMessageChannelBinder.java:282)
    at org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder.doBindConsumer(KafkaMessageChannelBinder.java:121)
    at org.springframework.cloud.stream.binder.AbstractBinder.bindConsumer(AbstractBinder.java:177)
    at org.springframework.cloud.stream.binding.ChannelBindingService.bindConsumer(ChannelBindingService.java:93)
    at org.springframework.cloud.stream.binding.BindableProxyFactory.bindInputs(BindableProxyFactory.java:189)
    at org.springframework.cloud.stream.binding.InputBindingLifecycle.start(InputBindingLifecycle.java:55)
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
    ... 27 more

The first log said I am using localhost:0.0.0.0:2181 for zookeeper connection, which I am not, and I have configured the real IP address for kafka.

the second log said the timeout for connection is 10000, which in kafka, I set the connection timeout to 6000

So these are not from kafka, then they are from scdf.

the fact is, when I look up the reference of scdf link, here it says, the zookeeper is not used (or maybe not directly used) in this architecture.

which confused me... if these logs not from kafka and not from scdf, then where does them from?

If scdf does use zookeeper, then how to configure the connection for zookeeper?

UPDATE

I am thinking about when scdf is trying to create a topic, from what I know about creating a topic you have to specify the zookeeper connection, scdf uses localhost for producers or consumers to register on zookeeper, so scdf is actually using zookeeper somehow. but how to configure this?

Answer 1

OK, after a dig through the reference file. I got this:

--spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.brokers=192.168.1.100:9092
--spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.zkNodes=192.168.1.100:2181

Hope this will help the ones who are looking for the answer....