ReplicatedMap EntryListener event on Split Brain Merge

Question

I use a ReplicatedMap<String, String> in my Hazelcast cluster. Each node has its key (hazelcastInstance.getLocalEndpoint().getUuid()) and some data (JSON string) in this map. Using this map I implemented a service such that each node has its data store, can access data of other nodes, and observe changes.

However I struggle to get the split brain behaviour right.

I would like to receive an entryAdded or entryUpdated event on my EntryListener attached to the ReplicatedMap when a split brain merge happens. However, this is not the case. I assume this is by design (?).

What solutions could you imagine to solve this?

Ideas:

• MembershipListener can detect the split brain merge. Problem: The data in ReplicatedMap is not necessarily ready yet - it could be ready before or after the memberAdded event is called.

• Implement MergePolicy, e.g.:

  public Object merge(String mapName, ReplicatedMapEntryView mergingEntry, ReplicatedMapEntryView existingEntry) {
      // every node knows its own values! (somehow existingEntry and mergingEntry are swapped?)
      // mergingEntry seems to be the destination (current local value)
      // existingEntry seems to be the value from the other member
      if (mergingEntry.getKey().equals(hazelcastUuid)) {
          return mergingEntry.getValue();
      } else {
          return existingEntry.getValue();
      }
  }
  

  Problem: This works fine when split brain merging two nodes. However, using four nodes (two nodes per machine on two different machines), it annihilates values and seems to behave randomly. I suspect that I understood something generally wrong because this approach is not working as hoped. Because of this, I didn't analyze the behavior of the four nodes yet - also it's quite time consuming to analyze four log files.