Uncaught exception 'MongoCursorException' with message 'Couldn't get connection: No candidate servers found'

Question

Problem Description

I have a three member replica set, and a php web front end that a) writes a record, and then b) does a .find() on the collection and returns all documents in the database.

To better understand how replica sets work, I did the following:

1. stopped the mongo service on the primary server(mongohost1). the web page kept working.
2. stopped the mongo service on the server that got promoted to primary (mongohost2). At this point, even though I have another mongo host (mongohost3) with the same database, the PHP web app fails with the above error message.

I was expecting that the system would let me at least read the records from the database, even if the write failed.

What I've checked / tried so far:

All of the hosts are reachable. I've trying pinging by hostname from each box and it alll works.

Here's how the replica set has been configured as per mongohost3:

        jlrs0:SECONDARY> cfg=rs.config()
        {
            "_id" : "jlrs0",
            "version" : 5,
            "members" : [
                {
                    "_id" : 0,
                    "host" : "monghost1.test.mm.org:27017",
                    "priority" : 3
                },
                {
                    "_id" : 1,
                    "host" : "mongohost2.test.mm.org:27017",
                    "priority" : 2
                },
                {
                    "_id" : 2,
                    "host" : "mongohost3.test.mm.org:27017",
                    "priority" : 2
                }
            ]
        }
        jlrs0:SECONDARY> 

and the status of each member in the replica set per mongohost3:

jlrs0:SECONDARY> rs.status()
{
    "set" : "jlrs0",
    "date" : ISODate("2014-11-19T15:16:21Z"),
    "myState" : 2,
    "members" : [
        {
            "_id" : 0,
            "name" : "mongohost1.test.mm.org:27017",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : Timestamp(1416419914, 1),
            "optimeDate" : ISODate("2014-11-19T17:58:34Z"),
            "lastHeartbeat" : ISODate("2014-11-19T15:16:20Z"),
            "lastHeartbeatRecv" : ISODate("2014-11-19T14:06:49Z"),
            "pingMs" : 0
        },
        {
            "_id" : 1,
            "name" : "mongohost2.test.mm.org:27017",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : Timestamp(1416419914, 5),
            "optimeDate" : ISODate("2014-11-19T17:58:34Z"),
            "lastHeartbeat" : ISODate("2014-11-19T15:16:17Z"),
            "lastHeartbeatRecv" : ISODate("2014-11-19T14:10:58Z"),
            "pingMs" : 0
    },
    {
        "_id" : 2,
        "name" : "mongohost3.test.mm.org:27017",
        "health" : 1,
        "state" : 2,
        "stateStr" : "SECONDARY",
        "uptime" : 451417,
        "optime" : Timestamp(1416419914, 5),
        "optimeDate" : ISODate("2014-11-19T17:58:34Z"),
        "self" : true
    }
],
"ok" : 1
}

Here's the PHP code to connect:

        $m = new   MongoClient("mongodb://mongohost1.test.mm.org:27017,mongohost2.test.mm.org:27017,mongohost3.test.mm.org:27017/?replicaSet=jlrs0");

I'm still reading up on replica sets etc. so I'm sure it's something that I've missed / neglected to set up. For example, I haven't set up an arbitor... not sure if it's related or not but just in case, i thought i'd mention it. I'm not sure what else to check.

Thanks.

Answer 1

You need to set your read preference to primaryPreferred. You need to specify that it ok to read from secondary when primary is not available. By default, it is not so.

Link to documentation

Answer 2

Please check also your php mongo pecl lib version. Before 1.5.6 there were 2 errors related to not selecting primary server by PHP after fail in replica set. Be sure to have pecl mongo at least 1.5.6.