Using RedisSentinelConfiguration doesn't fail over to the next available sentinel/master

Question

I'm trying to setup HA for my redis instances using redis-sentinel. I assume my java application has to be using RedisSentinelConfiguration for creating a connection factory. Hence I have setup the connection factory like below

@Bean(name = "redisConnectionFactory")
    public JedisConnectionFactory redisConnectionFactory() {
        String redisHost = redisSystemProperties.getProperty("redis.host");
        int redisPort = redisSystemProperties.getIntegerProperty("redis.port", DEFAULT_REDIS_PORT);
        logger.info(format("Creating redis connection factory.  host: %s, port: %d", redisHost, redisPort));
        JedisConnectionFactory connectionFactory = new JedisConnectionFactory(redisSentinelConfig(), jedisPoolConfig());
        boolean usePool = Boolean.valueOf(redisSystemProperties.getProperty("pool.enabled", "true"));
        connectionFactory.setUsePool(usePool);
        connectionFactory.afterPropertiesSet();
        return connectionFactory;
    }

    @Bean(name ="redisSentinelConfig")
    public RedisSentinelConfiguration redisSentinelConfig(){
        RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration();
        for (int i = 0; i < MAX_REDIS_MASTER_NODES; i++) {
            String hostPropertyName = format("redis.%d.host", i);
            String host = redisSystemProperties().getProperty(hostPropertyName);
            String portPropertyName = format("redis.%d.port", i);
            String portStr = redisSystemProperties().getProperty(portPropertyName);
            if (StrUtils.anyBlank(host, portStr)) {
                break;
            }
            int port = toInt(portStr);
            Assert.isTrue(port > 0, format("Invalid value '%s' for property '%s'", portStr, portPropertyName));
            RedisNode redisNode = new RedisNode(host, port);
            logger.info(format("Adding connection to redis sentinel [%d]: host: '%s', port: %d", i, host, port));
            sentinelConfig.addSentinel(redisNode);
            sentinelConfig.setMaster("mymaster");
        }
        return  sentinelConfig;
    }

The IP and port numbers that are used to create RedisSentinelConfiguration are of the sentinel instances I have started. I have started a sentinel for each master. I have 3 masters running so does 3 sentinels. The application seems to be working great as long as the master node which is mapped to sentinel 'mymaster' is running. Once I SHUTDOWN that redis node the application fails to get the connection. I see connection refused error. While debugging I can see that it is trying to create a connection using the single Sentinel configuration that it used in the first attempt.

Here is my sentinel configuration looks like.

port 26379

sentinel monitor mymaster 127.0.0.1 7000 2
sentinel down-after-milliseconds mymaster 1
sentinel failover-timeout mymaster 1
sentinel config-epoch mymaster 0
# Generated by CONFIG REWRITE
dir "/Users/skandula/redis-3.0.2"
sentinel leader-epoch mymaster 0
sentinel known-sentinel mymaster 127.0.0.1 26380 e4bc16a7435eec64512acc03404beb9799dea73e
sentinel known-sentinel mymaster 127.0.0.1 26381 493d598ce30dd4429bdc53c94cf297b8a0436c67
sentinel current-epoch 0

Can someone let me know what could I be doing wrong? I'm using spring-data-redis 1.5.0.RELEASE and Jedis-2.7.0

Thank you

Answer 1

There's a gap between failover and the actual implementation to pick up the newly elected master node. Although Sentinel itself might be configured to failover immediately, the java driver implementation will likely be a bit behind. You'll need to take care of those situations.

Running the sentinel example you can easily explore the behavior by randomly shutting down Redis and/or Sentinel instances, while watching the command line printing out errors and how long recovery actually took.

You could also use spring-retry to reschedule attempts of writing to redis. There's a ticket (DATAREDIS-370) open for integrating RetryTemplate - so you might want to vote for it.