RedisResponseException: Unknown reply on multi-request

Question

We have a windows service that runs a quartz job every minute to process reviews that were submitted more than 3 hours ago. The application uses the latest ServiceStack.Redis v3 library to interface with a Redis 2.8.12 instance on another machine.

When a new review is submitted, the ID of the new review is stored in a sorted set in Redis and we are using NewReview.DateCreated.Ticks for the score. When the job runs, it executes the following code in order to get the list of reviews to be processed:

using (var redisClient = RedisClientManager.GetClient())
{
    ...
    var cutOff = DateTime.Now.AddHours(-3);
    redisClient.GetRangeFromSortedSetByLowestScore("pending_reviews", 0L, cutOff);
    ...
}

Normally, this works fine and if there are any reviews in the sorted set 3 hours old or older their IDs are returned and the job processes them normally. However, the same exact code will intermittently cause the following exception to occur:

ServiceStack.Redis.RedisResponseException: Unknown reply on multi-request: ...
at ServiceStack.Redis.RedisNativeClient.CreateResponseError(String error)
at ServiceStack.Redis.RedisNativeClient.ReadMultiData()
at ServiceStack.Redis.RedisNativeClient.SendExpectMultiData(Byte[][] cmdWithBinaryArgs)
at ServiceStack.Redis.RedisNativeClient.GetRangeByScore(Byte[] commandBytes, String setId, Int64 min, Int64 max, Nullable`1 skip, Nullable`1 take, Boolean withScores)
at ServiceStack.Redis.RedisNativeClient.ZRangeByScore(String setId, Int64 min, Int64 max, Nullable`1 skip, Nullable`1 take)
at ServiceStack.Redis.RedisClient.GetRangeFromSortedSetByLowestScore(String setId, Int64 fromScore, Int64 toScore, Nullable`1 skip, Nullable`1 take)
at ServiceStack.Redis.RedisClient.GetRangeFromSortedSetByLowestScore(String setId, Int64 fromScore, Int64 toScore)

I tried downloading and stepping into the ServiceStack source code but the issue never happens when I'm debugging and I cannot seem to reproduce it otherwise. The RedisClientManager is a singleton PooledRedisClientManager and as far as I can tell I am creating and disposing of the client properly and I am not using a transaction or pipeline either.

My understanding is that the pooled client manager is doing some tricky connection sharing stuff since Redis is actually single-threaded. It feels like it might be returning the wrong results from another connection or some other threading or connection sharing issue.

Any ideas on what could be causing this?

Answer 1

Ok I figured it out.

Later on in the job, after the ids are retrieved, it was in fact using a transaction. When I reviewed the transaction code more closely, I realized that multiple client calls were being made within the context of a single QueueCommand. Although no errors occurred during the initial job invocation, the very next time the job ran it would always get the error.

So I simply broke each client call out into its own QueueCommand and Voila, the error went away. Lesson learned: when using transactions/pipelines be very careful that each redis client call is in it's own dedicated QueueCommand! In my case, it was hidden away in a helper method and I had to dig a bit to find it.