ReplyError: MOVED error after connecting to Redis Cluster AWS

Question

I am building a nodejs app that connects to redis. I have this working with my local instance of redis. Now, I am using ioredis to connect from my nodejs app to my redis cluster in k8s in AWS. Here is what I have.

const Redis = require("ioredis");

this.redis = new Redis({
 port: 6379, // Redis port
 host: rhost, // Redis host
 password: password
});

this.redis.on('connect', () => {
 console.log('connected to redis')
})

It seems like I successfully connect to my cluster as the message connected to redis prints out in the log. However, every time I try to use my redis object I get a MOVED error:

UnhandledPromiseRejectionWarning: ReplyError: MOVED 5011 <ip address>:6379
  at parseError (/node_modules/ioredis/node_modules/redis-parser/lib/parser.js:179:12)
  at parseType (/node_modules/ioredis/node_modules/redis-parser/lib/parser.js:302:14)

The connection works on my local. However, in AWS it does not. I tried swapping using the Redis.Cluster object instead of Redis, but after I deploy the app, the application hangs and the connection event never fires. The close and reconnecting events seem to be looping infinitely.

From my understanding, this is a problem with redirecting between nodes in the cluster. Perhaps it's a problem with the master/slave configuration. Is the error a configuration issue in AWS? Do I need to use the Redis.Cluster object instead of the plain Redis instance? What is the best way to fix the MOVED error?

Answer 1

The MOVED error is caused by using the Redis client directly and the configuration endpoint of ElastiCache (Redis Cluster Mode). This can be solved by using new Redis.Cluster([{ host: <cfg-endpoint>]) which is aware about multiple shards.

Otherwise disable the Redis Cluster Mode and use the primary (master) dns name of your ElastiCache cluster. Even without Redis Cluster Mode there is still a Failover Strategy (AWS will replace primary node when it fails) and you can deploy Nodes into multiple Availablity Zones.

Also when you have Encryption enabled you will need to connect to AWS ElastiCache with TLS enabled (tls: {} option for ioredis).

https://aws.amazon.com/blogs/database/work-with-cluster-mode-on-amazon-elasticache-for-redis/

Answer 2

I was also facing same issue as I used new Redis() instance which is not compatible with Redis Cluster Mode.

To make it compatible with Redis cluster, please try with below configuration,

//import section
  import { Cluster, Redis } from 'ioredis';

// connection config
   const connection = new Cluster([{
          host: process.env.host,
          port: parseInt(process.env.port),
        }], {
          slotsRefreshTimeout: 2000,
          // dnsLookup: This is needed when the addresses of startup nodes are hostnames instead of IPs.
          dnsLookup: (address, callback) => callback(null, address),
          redisOptions: {
            username: process.env.username,
            password: process.env.password,
            tls: {},
          }
    });

Above solution is tried and tested!

I am using ioredis package with latest version(5.3.2).

Answer 3

You can read more about the MOVE error here: https://redis.io/topics/cluster-spec#moved-redirection "MOVED Redirection A Redis client is free to send queries to every node in the cluster, including slave nodes. The node will analyze the query, and if it is acceptable (that is, only a single key is mentioned in the query, or the multiple keys mentioned are all to the same hash slot) it will lookup what node is responsible for the hash slot where the key or keys belong.

If the hash slot is served by the node, the query is simply processed, otherwise the node will check its internal hash slot to node map, and will reply to the client with a MOVED error, like in the following example:

GET x -MOVED 3999 127.0.0.1:6381 The error includes the hash slot of the key (3999) and the ip:port of the instance that can serve the query. The client needs to reissue the query to the specified node's IP address and port. Note that even if the client waits a long time before reissuing the query, and in the meantime the cluster configuration changed, the destination node will reply again with a MOVED error if the hash slot 3999 is now served by another node. The same happens if the contacted node had no updated information."

ioredis supports redis cluster. So you should be creating the redis cluster like so:

new Redis.Cluster([{
    host: process.env.REDIS_ENDPOINT, // Configuration endpoint address from Elasticache
    port: process.env.REDIS_PORT
  }]);

ioredis will take care of MOVE redirection errors.

Answer 4

If this doesn't help:

const Redis = require('ioredis');

const ioCluster = new Redis.Cluster([redisConfig.redis]);
const slackQueue = new Queue('slack notifications', {
  prefix: '{slack}' ,
  createClient: () => ioCluster
});
const emailQueue = new Queue('email notifications', {
  prefix: '{email}' ,
  createClient: () => ioCluster
});

I would go without ioredis or try to downgrade redis engine to 4.x

Answer 5

const Redis = require("ioredis");
const connection = await new Redis.Cluster(
        [redisHost,redisPort],
        {
      scaleReads: "all",
      redisOptions: {
        username: redisUserName,
        password: redisPassword,
        enableAutoPipelining: true,
      }
      );

The above configurations solves my issue, and I'm using ioredis insted of 'redis'.