How to recover from a terminated pulsar bookie

Question

We are running Apache Pulsar 2.72. in Prod which uses a 5 node (aws r5ad.2xlarge) bookie cluster (4.12.0). One of the nodes was terminated. As per our ASG a new node came up and joined the cluster. The Bookies have

1. autoRecoveryDaemonEnabled=true
2. lostBookieRecoveryDelay=0
3. bookkeeperClientMinNumRacksPerWriteQuorum=2
4. managedLedgerDefaultEnsembleSize=3
5. managedLedgerDefaultWriteQuorum=3

However the ledger re-replication wasn't taking place. I tried decommissioning the terminated node using sudo /opt/apache-pulsar/apache-pulsar-2.7.2/bin/bookkeeper shell decommissionbookie -bookieid bookieIP:port but it was stuck at 23:53:36.465 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 793 00:03:37.293 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 793 00:13:38.119 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 793 00:23:39.194 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 793 00:33:39.995 [main] INFO org.apache.bookkeeper.client.BookKeeperAdmin - Count of Ledgers which need to be rereplicated: 793 for more than 30 mins. We even tried getting the ledgers which were underreplicated using sh bookkeeper shell listunderreplicated and read some of the returned ledgers using sh bookkeeper shell ledger -m but that failed with an exception complaining about unable to access terminated bookie. We ended up deleting the underreplicated ledgers.

I am looking for a suggestion to best recover from a terminated bookie with our having to delete ledgers

Answer 1

Now that Apache Pulsar 2.8.1 is out, can you upgrade and try again. It seems unusual.

To get access to all the Pulsar people in one location, sign up for the summit

https://streamnative.io/en/blog/community/2021-09-07-speakers-announced-for-pulsar-virtual-summit-europe-2021/