Redis hooking (publish-subscribe) under stress tests - performance under load

Question

Based on the suggested solusion and following the example, I'm trying to delete a key right after I get a notification that another key has expired.

The problem is that under stress test with heavy load of seting 600K new keys and setting half of them with expiration time of 2 seconds, I get the following exception:

Exception in thread "main" redis.clients.jedis.exceptions.JedisConnectionException: Unknown reply: t

The question is what will be the best practice to write such listener? (thread pool? if so in what context to implement it?)

Jedis version: 2.7.2

Redis version: 2.8.19

---

My code so far:

Subscriber class:

public class Subscriber {

    public static void main(String[] args) {
        JedisPool pool = new JedisPool(new JedisPoolConfig(), "localhost");

        Jedis jedis = pool.getResource();
        jedis.psubscribe(new KeyExpiredListener(), "__key*__:*");

    }

}

Listener class:

public class KeyExpiredListener extends JedisPubSub {

    private String generalKeyTimeoutPrefix = "TO_";

    @Override
    public void onPMessage(String pattern, String channel, String message) {
        String originalKey = null;
        try {
            if(channel.endsWith("__keyevent@0__:expired") && message.startsWith(generalKeyTimeoutPrefix)) {
                originalKey = message.substring(generalKeyTimeoutPrefix.length());
                del(originalKey);
            }
        } catch(Exception e) {
            logger.error("..", e);
        }
    }

    private void del(String key) {
        Jedis jedis = new Jedis("localhost");
        jedis.connect();
        try {
            jedis.del(key);
        } catch (Exception e) {
            logger.error("...");
        } finally {
            jedis.disconnect();
        }
    }
}

Key generator:

public class TestJedis {
    public static void main(String[] args) throws InterruptedException {
        JedisPool pool = new JedisPool(new JedisPoolConfig(), "localhost");
        Jedis jedis = pool.getResource();
        String prefixForlKeys = "token_";
        String prefixForTimeoutKeys = "TO_";
        for (int i = 0; i < 300000; i++) {
            String key = prefixForlKeys + i;
            String timeoutKey = prefixForTimeoutKeys + key;
            jedis.set(key, "some_data");
            jedis.set(timeoutKey, "");
            jedis.expire(timeoutKey, 2);
        }
      System.out.println("Finished to create the keys");  
    }
}

Answer 1

the problem is with your implementation of del() method: it does not use connection pooling, does not reuse connections, so it finally occupies all available local ports. Try to use something like this:

private void del(String key) {
    Jedis jedis = pool.getResource();
    jedis.connect();
    try {
        jedis.del(key);
    } catch (Exception e) {
        e.printStackTrace(  );
    } finally {
        jedis.close();
    }
}

instead of opening/closing a connection for each expired key.