We are using spark standalone cluster 3 zookeepers in HA mode.I am seeing this issue in zookeeper.log.
Exception causing close of session 0x0 due to java.io.IOException: Len error 1195725856 Closed socket connection for client /10.23...... (no session established for client)
Zookeeper leader is getting auto failed over from one server to another server and hence followed by this, spark master is getting auto failed over Also Some clients are getting continually disconnected/reconnected with this error. How to fix
Full Log:
[myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@354] - Exception causing close of session 0x0 due to java.io.IOException: Len error 1195725856
[myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /10....... (no session established for client)
[myid:3] - ERROR [LearnerHandler-/159.1.......:LearnerHandler@562] - Unexpected exception causing shutdown while sock still open
[myid:3] - WARN [LearnerHandler-/159.1......:LearnerHandler@575] - ******* GOODBYE /159.1..... ********
[myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x29000000ed (n.zxid), 0xa (n.round), LOOKING (n.state), 1 (n.sid), 0x29 (n.peerEPoch), LEADING (my state)
[myid:3] - INFO [LearnerHandler-/159........:LearnerHandler@263] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@e144103
[myid:3] - INFO [LearnerHandler-/159.......LearnerHandler@318] - Synchronizing with Follower sid: 1 maxCommittedLog=0x29000000ed minCommittedLog=0x2800000007 peerLastZxid=0x29000000ed
[myid:3] - INFO [LearnerHandler-/159.......:LearnerHandler@395] - Sending DIFF