How to recover client from "No handler waiting for message" warning?

Question

At medium to high load (test and production), when using the Vert.x Redis client, I get the following warning after a few hundred requests.

2019-11-22 11:30:02.320 [vert.x-eventloop-thread-1] WARN io.vertx.redis.client.impl.RedisClient - No handler waiting for message: [null, 400992, <data from redis>]

As a result, the handler supplied to the Redis call (see below) does not get called and the incoming request times out.

Handler<AsyncResult<String>> handler = res -> {
    // success handler
};

redis.get(key, res -> {
    handler.handle(res);
});

The real issue is that once the "No handler ..." warning comes up, the Redis client becomes useless because all further calls to Redis made via the client fails with the same warning resulting in the handler not getting called. I have an exception handler set on the client to attempt reconnection, but I do not see any reconnections being attempted.

How can one recover from this problem? Any workarounds to alleviate the severity would also be great.

I'm on vertx-core and vertx-redis-client 3.8.1 .

Answer 1

The upcoming 4.0 release had addressed this issue and a release should be hapening soon, how soon, I can't really tell.

The problem is that we can't easily port back from the master branch to the 3.8 branch because a major refactoring has happened on the client and the codebases are very different.

The new code, uses a connection pool and has been tested for concurrent access (and this is where the issue you're seeing comes from). Under load the requests are routed across all event loops and the queue that maintains the state between in flight requests (requests sent to redis) and waiting handlers would get out of sync in very special conditions.

So I'd first try to see if you can already start moving your code to 4.0, you can have a try with the 4.0.0-milestone3 version but to be totally fine, just have a run with the latest master which has more issues solved in this area.