What Redis commands don't work in cluster mode?

Question

UPDATE - 2023 - Verx Redis now supports scan but there's a bug see https://github.com/vert-x3/vertx-redis-client/issues/345

My team has been using the Vertx Redis client for some time and using the keys command with a Redis cluster and I was asked to refactor to use the scan command instead since "keys" isn't meant for production.

When I went to use scan I got an error from the client saying "scan not supported - use non cluster client on the right node." So it appears that the Vertx Redis client supports cluster-wide "keys" but not "scan." I learned that I'm supposed to create 1 client per node if I want to do a cluster-wide "scan" e.g. something like this but that seems like an overfly complex and poorly performant choice (or I can use "hash tags" which won't work for my use case).

I'm trying to figure out which commands are supported cluster-wide and which aren't (so we don't make another mistake like this ) and the only hint I see is in the Redis cluster spec:

Redis Cluster implements all the single key commands available in the non-distributed version of Redis.

My question is - what is a single key command?

Or another explanation I saw here was the following:

The only different between distributed and non-distributed Redis clients is that in the distributed case MOVED and ASKS will be "followed."

I'm also not clear on how a MOVED or ASK is followed - I assume it just means that the client reconnect to the correct node and tries again? That seems horribly non-performant as discussed here.

It's also odd that the Vertx client documentation doesn't mention this or the more popular Jedis client.

Am I missing some key documentation that explains all this?

Answer 1

My question is - what is a single key command?

It's a command, which has a single key a as parameter. For example, SET key value, only sets a single key.

In fact, you missed the second sentence: Commands performing complex multi-key operations ... are implemented for cases where all of the keys ... hash to the same slot. So if k1, k2 and k3 are located on the same hash slot, you can send MGET k1 k2 k3 command to Redis Cluster.

I assume it just means that the client reconnect to the correct node and tries again?

YES.

That seems horribly non-performant as discussed here.

A decent Redis client should cache the slot-node mapping. So if the slot-node mapping has not changed, clients can calculate the right slot/node based on key, and send the command to the right node. So that it can avoid the redirection.

Answer 2

To understand why scan has not been implemented in Vert.x Redis Cluster Client we first need to know how SCAN command works. Redis SCAN

In the standalone mode what you would do is for example scan for all keys matching a pattern on the node. As per documentation we would initiate the first call with SCAN 0 MATCH *11* where 0 is the beginning cursor. As a result we would get something like this:

1) "288"
2) 1) "key:911"
   2) "key:811"
   3) "4511"
   4) "some11key"

After this result you would call the next SCAN with returned cursor like SCAN 288 MATCH *11* and so on until you get returned cursor 0. So on a standalone Redis Server you would probably call SCAN command repeatedly in while loop most probably until you get a response with cursor 0 which marks end of the SCAN.

Implementing this on a Cluster Client is practically impossible as the client would need to initiate the command on all nodes with different hash slots and return to you multiple response unique to each node. After that it is impossible to continue as you would get a different cursor for each node. So client would need to do some background magic and keep the state in order to continue. So this is not a limitation of Vert.x Redis Client but more of a limitation from SCAN command works. The only workaround is perform the command fully from start to finish for each different node manually.

My question is - what is a single key command?

Single key commands are commands that work with only one key. This is to play it safe and say commands that operate only on one node will 100% work out of the box without any special client implementation. This does not prevent the client implementations from doing some additional work in order to make multi key commands work like for example KEYS command. You can see this example and few more commands that have been implemented to work with Redis Cluster although they are multi key commands. KEYS implementation Vert.x Redis Cluster Client

I'm also not clear on how a MOVED or ASK is followed

When cluster topology changes for any reason client is most probably left with incorrect hash slot status. So if you initiate a command on the cluster client with incorrect hash slot data it will try and send the request to the wrong server. In this case server will respond with MOVE/ASK response and client should follow it in order to send the command to the correct server. Also because of the performance it is only sane to refresh the client hash data so that we don't send commands to the wrong server again. And this is exactly how Vert.x Redis Client is implemented. Some other clients also have background periodic check in order to keep the hash slot data updated. This is one of the features that is planned for Vert.x Redis Cluster Client.

Answer 3

Since my original question wasn't answered let me try after further research:

What Redis commands don't work in cluster mode?

None - all of the non-clustered commands will run and in some sense "work" on a Cluster.

Where is the documentation that clarifies how different commands work in a Cluster vs standalone?

It is here:

https://redis.io/docs/reference/command-tips/#request_policy https://redis.io/docs/reference/command-tips/#response_policy

Basically there are 3 options when running a command against a cluster

1. Execute it on a random node
2. Execute it on all nodes
3. The "special" non-trivial case

What Redis commands are likely to confuse developers when run in a cluster?

The only command that appears to behave in an especially confusing way is SCAN. Which is why it is the only command to be flagged as being ["request_policy:special" by Redis][1]

The reason is very straightforward from the doc. Basically some commands can be executed on any node and some commands should be executed on all nodes e.g. in a for loop where you aggregate the responses.

The problem with a Cluster SCAN is that it doesn't make sense to execute it on only 1 node or all nodes hence it's a special case. The reason is that SCAN returns a cursor and you may need to iterate a different number of times on each node. E.g. you may have a cluster wide scan that requires 1 SCAN on NODEA but 3 SCANs on NODEB.