Milvus pulsar bookies relation to S3 (Minio)

Question

I have deployed Milvus cluster 2.2.2 in k8s and after some hardware problem two of my three pulsar bookies have their pv/pvc corrupted and failing to start.

I see there is 50 ledgers underreplicated in the healthy bookie1 but when I try to execute:

 bookie1# booker shell recover bookie2(failed)
 bookie1# booker shell recover bookie3(failed)

there is errors.

So my question: I have loaded the data completely few months ago and I see it in the S3/Minio storage, but can I delete the ledgers and start the bookies from 0, will this affect data availability if the data inside the bookies is lost?

Error 1:

 09:43:22.800 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There 
 are directories without a cookie, and this is neither a new 
 environment, nor is storage expansion enabled. Empty directories are 
 [/pulsar/data/bookkeeper/journal/current]
 09:43:22.800 [main] INFO  
 org.apache.bookkeeper.proto.BookieNettyServer - Shutting down 
 BookieNettyServer
 09:43:22.829 [main] ERROR org.apache.bookkeeper.server.Main - Failed 
 to build bookie server
 org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
    at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:494) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
    at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:273) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
    at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:731) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
    at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
    at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
    at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
    at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
    at org.apache.bookkeeper.server.Main.doMain(Main.java:226) [org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]
    at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3]

Answer 1

From another thread: If all the pulsar messages were consumed by milvus, then you can relaunch a new/clean pulsar service and stop the old pulsar service. Change the pulsar address in milvus.yaml to the new pulsar service and restart the milvus cluster. Milvus will work well with the new pulsar service.