I have 5 ordering nodes (version 2.4) running on my cluster. I tried to rotate their tls certs and made some mistake. The network is a dev network and now all the ordering nodes are throwing below mentioned error:
2022-03-25 10:18:13.947 UTC 014c WARN [orderer.common.cluster.puller] fetchLastBlockSeq -> Received status:NOT_FOUND from orderer1.org1.com:7050: faulty node, received: status:NOT_FOUND channel=assetschannel
2022-03-25 10:18:13.947 UTC 014d WARN [orderer.common.cluster.puller] func1 -> Received error of type 'faulty node, received: status:NOT_FOUND ' from orderer1.org1.com:7050 channel=assetschannel
2022-03-25 10:18:13.948 UTC 014e WARN [orderer.common.cluster.puller] fetchLastBlockSeq -> Received status:NOT_FOUND from orderer4.org1.com:7050: faulty node, received: status:NOT_FOUND channel=assetschannel
2022-03-25 10:18:13.948 UTC 014f WARN [orderer.common.cluster.puller] func1 -> Received error of type 'faulty node, received: status:NOT_FOUND ' from orderer4.org1.com:7050 channel=assetschannel
2022-03-25 10:18:13.948 UTC 0150 WARN [orderer.common.cluster.puller] fetchLastBlockSeq -> Received status:NOT_FOUND from orderer2.org1.com:7050: faulty node, received: status:NOT_FOUND channel=assetschannel
2022-03-25 10:18:13.949 UTC 0151 WARN [orderer.common.cluster.puller] func1 -> Received error of type 'faulty node, received: status:NOT_FOUND ' from orderer2.org1.com:7050 channel=assetschannel
2022-03-25 10:18:13.949 UTC 0152 WARN [orderer.common.cluster.puller] fetchLastBlockSeq -> Received status:NOT_FOUND from orderer0.org1.com:7050: faulty node, received: status:NOT_FOUND channel=assetschannel
2022-03-25 10:18:13.949 UTC 0153 WARN [orderer.common.cluster.puller] func1 -> Received error of type 'faulty node, received: status:NOT_FOUND ' from orderer0.org1.com:7050 channel=assetschannel
2022-03-25 10:18:13.953 UTC 0154 WARN [orderer.common.cluster.puller] fetchLastBlockSeq -> Received status:NOT_FOUND from orderer3.org1.com:7050: faulty node, received: status:NOT_FOUND channel=assetschannel
2022-03-25 10:18:13.953 UTC 0155 WARN [orderer.common.cluster.puller] func1 -> Received error of type 'faulty node, received: status:NOT_FOUND ' from orderer3.org1.com:7050 channel=assetschannel
Since it is a dev network, I can reset the entire network but I want to consider this scenario as if it occurred on a production network and because we can not reset a production network, I would like to get some suggestions on how we can recover the ordering nodes from this error.
Any suggestions would be appreciated
As per my knowledge, If we have backup of the ledger and WAL, we can restore them & bring up the network. Otherwise, we need to reset the network.