Spring XD stream and job persistence

Question

I am very new to Spring XD and using xd-singlenode to run a Spring XD locally. I have created few streams and taps. I would like to know the following:

1. How can I persist stream and job definitions to a database so I do not need to recreate them whenever I restart the server (for local and distributed mode).

2. How should I implement a pub-sub mechanism to send large volumes of binary/text data to the requester using Spring XD.

Thanks!

Answer 1

For the first request, you need to use a separate ZooKeeper server (when you start XD in singlenode mode, it transparently creates an embedded ZooKeeper server for you. But you can point it to a distinct ZK, which you would have to do if you were running in distributed mode). Stream definitions are stored in ZooKeeper, so they'd survive a spring XD server restart. A typical setup for ZooKeeper is to use port 2181, so you could start XD singlenode like so (*nix):

ZK_CLIENT_CONNECT=localhost:2181 ./bin/xd-singlenode

(Or you can edit your servers.yml file, looking for the zk.client.connect key)

As for your second question, I'm not sure I understand it. Feel free to clarify

Answer 2

When you are using single node, streams and jobs are not persistent, you can use distributed node by starting the following servers: 1. Redis 2. Hsqldb 3. container 4. admin

Then you'll not need to recreate your streams and jobs each time.

Moha.