Detecting killed nodes with Akka Cluster

Question

I'm working on a project using Akka Cluster 2.4.8.

Is there a way to detect crashed nodes (as in computer failure, kill -9, etc) with AkkCluster ?

I'm currently have a 3-node environment, using the static-quorum split brain resolve strategy.

akka.cluster.split-brain-resolver {
    active-strategy = static-quorum
    stable-after = 5s

    static-quorum {
        quorum-size = 2
        role = ""
    } 

I was hoping that when killing an instance, the remaining cluster members would mark it as DOWN. However, it remains UNREACHABLE (see below). Is there a way to achieve this?

clusterStatus": {
    "members": [
        {
            "uniqueAddress": {
                "address": {
                    "protocol": "akka.tcp",
                    "system": "test-actor-system",
                    "host": "test-out-00",
                    "port": 2552
                },
                "uid": 1998600863
            },
            "upNumber": 1,
            "status": "Up",
            "roles": []
        },
        {
            "uniqueAddress": {
                "address": {
                    "protocol": "akka.tcp",
                    "system": "test-actor-system",
                    "host": "test-out-01",
                    "port": 2552
                },
                "uid": 1371217592
            },
            "upNumber": 3,
            "status": "Up",
            "roles": []
        },
        {
            "uniqueAddress": {
                "address": {
                    "protocol": "akka.tcp",
                    "system": "test-actor-system",
                    "host": "test-out-02",
                    "port": 2552
                },
                "uid": -796176254
            },
            "upNumber": 2,
            "status": "Up",
            "roles": []
        }
    ],
    "unreachable": [
        {
            "uniqueAddress": {
                "address": {
                    "protocol": "akka.tcp",
                    "system": "test-actor-system",
                    "host": "test-out-01",
                    "port": 2552
                },
                "uid": 1371217592
            },
            "upNumber": 3,
            "status": "Up",
            "roles": []
        }
    ]

Answer 1

Split-brain resolver is a commercial feature of Akka for which you need a Lightbend subscription.

Note

This is a feature of the Typesafe Reactive Platform that is exclusively available for Typesafe Project Success Subscription customers.

To use the Split Brain Resolver feature you must install Typesafe Reactive Platform.

Chances are, if you're not a Reactive Platform subscriber, your split-brain configs are just plainly ignored.

Full docs at http://doc.akka.io/docs/akka/rp-15v09p02/scala/split-brain-resolver.html

Answer 2

To be honest I'm not using Akka 2.4+. I've started my project with akka 2.3.12 and I still use it. At that time there was no split-brain plugin solution provided by akka, the only advice was to set:

# put to off in order to not have split brain
auto-down-unreachable-after = off

This meant, in order to avoid split brain, you had to manual remove UNREACHABLE nodes, and back in 2.3.12 the akka-microkernel (that was later deprecated http://doc.akka.io/docs/akka/2.4.1/project/migration-guide-2.3.x-2.4.x.html#Microkernel_is_Deprecated) gave you the possibility to issue a command to the cluster to mark the node with issues as DOWN

bin/akka-cluster localhost 9999 down akka.tcp://MySystem@darkstar:2552

So a bit of user-needed action to manage the split-brain and to take a node out of the cluster, it may be the same case with your (Akka Cluster 2.4.8) version.