2 Node Redis HA

Question

I have two nodes which I want to run as servers in active-active mode and also have HA capability i.e if one is down, the other one should start receiving all the requests but while both are up, both should be taking all the requests. Now since Redis doesn't allow active-active mode for the same hash set and I don't have option to run Sentinel as I can't have a third node, my idea is to run the two nodes in replication and myself decide if master node is down and promote the slave to master. Are there any issues with this? When the original master comes back up, is there a way to configure it as slave?

Does this sound like a good idea? I am open to suggestions other than Redis.

Answer 1

Ok, partial solution with SLAVEOF:

You can manually promote slave to master by running:

SLAVEOF NO ONE

You can manually transition master to slave by running:

SLAVEOF <HOST> <port>

Clustering should be disabled.

Answer 2

Generally running two node is never a good idea because it is bound to have split brain problem: When the network between the two node is down for a moment or two, the two node will inevitably think each other is offline and will promote/keep itself to be master and start accepting requests from other services. Then the split brain happens.

And if you are OK with this possible situation, then you can look into setup a master-slave with help of a script file and a HA service like pacemaker or keepalived.

Typically you have to tell the cluster manager through a predefined rule that when two machine rejoins under split brain condition, which one is your preferred master.

When a master is elected, execute the script and basically it execute slaveof no one on itself and execute slaveof <new-master-ip> <port> on the other node.

You could go one step further in your script file and try to merge the two data sets together but whether that's achievable or not is entirely down to how you have organized your data in Redis and how long you are prepared to wait for to have all the data in sync.

I have done this way myself before through pacemaker+corosync.

Answer 3

I would recommendation to have at least 3 nodes with Sentinel Setup for enabling gossip/quorum for auto promotion of slave to master when current master node goes down.

Answer 4

Python example using redis module, note I haven't tested, but I think this should work:

import redis
import random

def get_redis_connection():
    # List of Redis nodes
    redis_nodes = [
        {'host': 'redis1.example.com', 'port': 6379},
        {'host': 'redis2.example.com', 'port': 6379}
    ]

    # Shuffle the nodes to distribute the load
    random.shuffle(redis_nodes)

    # Attempt connection to each node until successful or all fail
    for node in redis_nodes:
        try:
            r = redis.Redis(host=node['host'], port=node['port'])
            r.ping()  # Test the connection
            return r
        except redis.exceptions.ConnectionError:
            continue

    raise Exception("Failed to connect to any Redis node")

# Usage example
redis_connection = get_redis_connection()
redis_connection.get('key')

Answer 5

I believe it is possible to create a cluster with two nodes with the commands below:

$ redis-cli --cluster create <ip-node1>:7000 <ip-node1>:7001 <ip-node2>:7000 <ip-node2>:7001 --cluster-replicas 1

To resolve the split-brain problem. you can add a third node without data:

$ cluster meet #IP_node3#:7000

$ cluster nodes

I think it works.

Answer 6

If you brought the replica online manually by changing it to replicaof no one, you need to be careful to bring the failed master back online as a replicaof the new node so you dont overwrite more recent data. I would not recommend doing this manually. You want to minimize downtime so automated failover is ideal

You mention being open to other products. Check out KeyDB which has the exact configuration you are looking for. It is a maintained multi-threaded fork of redis which offers the active-replica scenario you are looking for. Check out an example of it here.

Run both nodes as replicas of each other accepting reads and writes simultaneously (depending on upfront proxy config). If one fails the other continues to take the full load and is already sync'd.

Regarding the split brain concern, KeyDB can handle split brain scenarios where the connection between masters is severed, but writes continue to be made. Each write is timestamped and when the connection is restored each master will share their new data. The newest write will win. This prevents stale data from overwriting new data written after the connection was severed.