Do consul possess an upper limit on number of consul watches?

Question

I am creating 200 watcher and i am getting EOF error repeatedly while Querying any consul CLI commands and then disconnection takes place. on consul logs i am seeing “too many files open” error while getting my consul up itself.

for ~150 entries I’m good to keep a watch and provide update on an HTTP endpoint.

Setup : 3 node consul agents in server mode were running. configuration file tries to register 200 Key watchers on KVs when one consul agent boots up.

I am doubting that this it is hitting open FD limit.

output of details are here :

1. ulimit Output

[rajkumar]# ulimit -a

• core file size (blocks, -c) 0
• data seg size (kbytes, -d) unlimited
• scheduling priority (-e) 0
• file size (blocks, -f) unlimited
• pending signals (-i) 31186
• max locked memory (kbytes, -l) 64
• max memory size (kbytes, -m) unlimited
• open files (-n) 16384
• pipe size (512 bytes, -p) 8
• POSIX message queues (bytes, -q) 819200
• real-time priority (-r) 0
• stack size (kbytes, -s) 8192
• cpu time (seconds, -t) unlimited
• max user processes (-u) 31186
• virtual memory (kbytes, -v) unlimited
• file locks (-x) unlimited

2. for lsof on consul grep gives these results

• before running an agent

[rajkumar]# lsof | grep ‘consul’ | grep ‘ESTABLISHED’ | wc -l

0

[rajkumar]# lsof | grep ‘consul’ | wc -l

603

• with 150 watches these results are:

[rajkumar]# lsof | grep ‘consul’ | grep ‘ESTABLISHED’ | wc -l

3750

[rajkumar]# lsof | grep ‘consul’ | wc -l

4563

• with 500 watches running

[rajkumar]# lsof | grep ‘consul’ | grep ‘ESTABLISHED’ | wc -l

4040

[rajkumar]# lsof | grep ‘consul’ | wc -l

4881

3. consul put response when the system stops responding

[rajkumar]# consul kv put new one Error! Failed writing data: Put http://127.0.0.1:8500/v1/kv/new: EOF

4. consul members

[rajkumar]# consul members

Error retrieving members: Get http://127.0.0.1:8500/v1/agent/members?segment=_all: EOF

5. consul get command error

[rajkumar]# consul kv get key14

Error querying Consul agent: Get http://127.0.0.1:8500/v1/kv/key14: read tcp 127.0.0.1:55574->127.0.0.1:8500: read: connection reset by peer

6. consul agent process logs after boot up. I have created 500 watches in the config file.