Recover MongoDB data from inaccessible replica set

Question

I had previously created a 3-node docker cluster of MongoDB with port 27017 mapped to that of respective hosts.

I had then created a replica set rs0 with it's members being host1.mydomain.com:27017, host2.mydomain.com:27017 and host3.mydomain.com:27017. Please note that while creating the replica set, I had specified members with their mydomain.com addresses and not with ${IP1}:27017, etc. I had the respective DNS records set up for each host.

Thus, I could connect to this cluster with string:

mongodb+srv://admin:<pass>@host1.mydomain.com,host2.mydomain.com,host3.mydomain/admin?replicaSet=rs0

Unfortunately, I have lost access to mydomain.com as it has expired and has been scooped up by another buyer.

I can still SSH into individual hosts and log into docker containers, type mongo, then use admin; and then successfully authenticate using db.auth(<user>, <pass>). However, I cannot connect to the replica set nor can export the data out of it.

Here's what I get if I try to SSH into one of the nodes and try to access the data:

$ mongo
MongoDB shell version v3.6.8
connecting to: mongodb://127.0.0.1:27017
Implicit session: session { "id" : UUID("fc3cf772-b437-47ab-8faf-5e0d16158ff0") }
MongoDB server version: 4.4.10

> use admin;
switched to db admin

> db.auth('admin', <pass>)
1

> show dbs;
2022-07-22T13:37:38.013+0000 E QUERY    [thread1] Error: listDatabases failed:{
    "topologyVersion" : {
        "processId" : ObjectId("62da79de34490970182aacee"),
        "counter" : NumberLong(1)
    },
    "ok" : 0,
    "errmsg" : "not master and slaveOk=false",
    "code" : 13435,
    "codeName" : "NotPrimaryNoSecondaryOk"
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
Mongo.prototype.getDBs@src/mongo/shell/mongo.js:67:1
shellHelper.show@src/mongo/shell/utils.js:860:19
shellHelper@src/mongo/shell/utils.js:750:15
@(shellhelp2):1:1

> rs.slaveOk();

> show dbs;
2022-07-22T13:38:04.016+0000 E QUERY    [thread1] Error: listDatabases failed:{
    "topologyVersion" : {
        "processId" : ObjectId("62da79de34490970182aacee"),
        "counter" : NumberLong(1)
    },
    "ok" : 0,
    "errmsg" : "node is not in primary or recovering state",
    "code" : 13436,
    "codeName" : "NotPrimaryOrSecondary"
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
Mongo.prototype.getDBs@src/mongo/shell/mongo.js:67:1
shellHelper.show@src/mongo/shell/utils.js:860:19
shellHelper@src/mongo/shell/utils.js:750:15
@(shellhelp2):1:1

How do I go about this? The DB contains important data that I would like to export or simply have the cluster (or one of the mongo hosts) running again.

Thanks!

Answer 1

Add following records to /etc/hosts file on each container running mongodb, and the client where you are connecting from:

xxx.xxx.xxx.xxx  host1.mydomain.com
yyy.yyy.yyy.yyy  host2.mydomain.com
zzz.zzz.zzz.zzz  host3.mydomain.com

replace xxx, yyy, zzz with actual IP addresses that listen on 27017.

If the client is Windows, the hosts file is located at %SystemRoot%\System32\drivers\etc\hosts

If the replica set restores, you will be able to connect to the database without +srv schema:

mongodb://admin:<pass>@host1.mydomain.com,host2.mydomain.com,host3.mydomain.com \
?authSource=admin&replicaSet=rs0

If you don't know network configuration, or the replica set did not recover for any reason, you still can connect to individual node as standalone instances.

Restart the mongodb without --replSet parameter in the command line (somewhere in your Dockerfile) or replication part in mongodb.conf. It will resolve the "NotPrimaryOrSecondary" error.