PRIMARY is transitioning to RECOVERING after restart

Question

I have a three-node replica set (1 Primary, 1 Secondary, 1 Arbiter), on three different Amazon server instances. The servers where they are hosted required a memory upgrade so I needed to shut down the MongoDB instances as well.

I shut down the MongoDB instances in this order:

1. Secondary
2. Arbiter
3. Primary

I used the process below for shutting down each server

use admin
db.shutdownServer() 

All MongoDB instances did shut down properly without any problems. So far everything is fine.

After the Amazon server upgrade, I started the MongoDB instances in the following order:

1. Arbiter
2. Secondary
3. Primary

The arbiter is in arbiter mode and secondary is in secondary mode, but to my surprise the primary machine went to "RECOVERING" mode.

I don't know the reason, why the primary machine went to "RECOVERING".

I have examined logs. It is showing no member to sync...something stuff like that

My basic doubt is "PRIMARY has to be in PRIMARY until there is reconfig happens in replica set".

Am I missing a step during the shutdown of servers? Or am I missing a step during the restart of servers?

Please shed some light on this so that how can I overcome this problem. I need to shut down the MongoDB servers frequently since there is a lot of upgrades happening in Amazon servers.

Answer 1

After you started you replica set, you "SECONDARY" became "PRIMARY" and you "PRIMARY" was probably at secondary state after short while. To keep primary status at your "PRIMARY", you must give it higher priority than what your "SECONDARY" have.

Check with rs.conf() command.

Check here how to force node to be primary