Spring boot (Redis) LettuceConnectionFactory unable to recover if any node goes down from cluster

118 views Asked by At

I have 4 node cluster machine

  1. 3 Replica of Redis
  2. 3 Replica of Sentinel

Deleting Redis master node LettuceConnectionFactory throws the following error and does not recover from this state and stops all other Redis operations.

Cannot connect Redis Sentinel at RedisURI [host='sentinel', port=5000]: java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeoutException: connection timed out: sentinel:5000
[2023-10-30 19:27:16,749] [ERROR] [lettuce-nioEventLoop-4-2] [io.lettuce.core.masterslave.SentinelConnector.lambda$null$3 : 116]
    Error during background refresh

            io.lettuce.core.RedisConnectionException: Unable to connect to RedisURI [sentinels=[RedisURI [host='sentinel', port=5000]], sentinelMasterId=mymaster]
        
        Caused by: io.lettuce.core.RedisConnectionException: Cannot connect Redis Sentinel at RedisURI [host='sentinel', port=5000]
            at io.lettuce.core.RedisClient.lambda$connectSentinelAsync$9(RedisClient.java:555)
            at reactor.core.publisher.Mono.lambda$onErrorMap$28(Mono.java:3763)
            at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
            ... 29 common frames omitted
    
    Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: sentinel:5000
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)

Following is my LettuceConnectionFactory connection

ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                .enablePeriodicRefresh(Duration.ofSeconds(1)) // Refresh the topology periodically.
                .enableAllAdaptiveRefreshTriggers() // Refresh the topology based on events.
                .build();

        ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
                .timeoutOptions(TimeoutOptions.enabled(Duration.ofSeconds(15)))
                .topologyRefreshOptions(topologyRefreshOptions).build();

        RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration().master("mymaster");
        sentinelConfig.sentinel(host, Integer.parseInt(port));

        return new LettuceConnectionFactory(sentinelConfig, LettuceClientConfiguration.builder()
                .clientOptions(clusterClientOptions).readFrom(ReadFrom.REPLICA_PREFERRED).build());

is there any parameter that i am missing?

0

There are 0 answers