In my web application i'm using JedisPool(v3.0.1) to generate unique transaction id across multiple instance. But After sometime we are getting JedisExhaustedPoolException even though when i print jedis pool stats it shows idle Connection=8. Also 8 Connections to redis are established when i check using client list command in redis-cli
Pool stats logger:
2024-01-16 17:20:20,635|http-nio-9020-exec-28|RedisConnectionPool|(getConnection - 52)|24011617202010371|RBT|Redis DB Stats:Idle 8, Waiting 0, Active 0
Full Exception stacktrace:
2024-01-16 17:20:20,642|http-nio-9020-exec-28|RequestSender|(addCustomParamDetailsIfExist - 790)|24011617202010371|RBT|Error while generating custom parameter
redis.clients.jedis.exceptions.JedisException: Error while updating counter sequence in Redis
at com.mcom.afl.db.DBUtility.updateRedisCounter(DBUtility.java:6809) ~[classes/:B-1-C-1]
at com.mcom.afl.util.RequestSender.generateResetRequestCounter(RequestSender.java:1118) ~[classes/:B-1-C-1]
at com.mcom.afl.util.RequestSender.getCustomValForOtherTypes(RequestSender.java:1003) ~[classes/:B-1-C-1]
at com.mcom.afl.util.RequestSender.getCustomParamValue(RequestSender.java:957) ~[classes/:B-1-C-1]
Caused by: redis.clients.jedis.exceptions.JedisExhaustedPoolException: Could not get a resource since the pool is exhausted
at redis.clients.jedis.util.Pool.getResource(Pool.java:53) ~[jedis-3.0.1.jar:?]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234) ~[jedis-3.0.1.jar:?]
at com.mcom.afl.db.RedisConnectionPool.getConnection(RedisConnectionPool.java:55) ~[classes/:B-1-C-1]
at com.mcom.afl.db.DBUtility.updateRedisCounter(DBUtility.java:6800) ~[classes/:B-1-C-1]
... 97 more
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:494) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.4.3.jar:2.4.3]
at redis.clients.jedis.util.Pool.getResource(Pool.java:50) ~[jedis-3.0.1.jar:?]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234) ~[jedis-3.0.1.jar:?]
at com.mcom.afl.db.RedisConnectionPool.getConnection(RedisConnectionPool.java:55) ~[classes/:B-1-C-1]
at com.mcom.afl.db.DBUtility.updateRedisCounter(DBUtility.java:6800) ~[classes/:B-1-C-1]
... 97 more
Client list command output: my application IP is 172.22.39.1
127.0.0.1:6379> client list
id=942 addr=172.22.39.1:54882 fd=15 name= age=25 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=943 addr=172.22.39.1:54884 fd=16 name= age=25 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=944 addr=172.22.39.1:54890 fd=17 name= age=25 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=945 addr=172.22.39.1:54892 fd=18 name= age=25 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=903 addr=127.0.0.1:42950 fd=19 name= age=130 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 qbuf-free=32742 obl=0 oll=0 omem=0 events=r cmd=client
id=937 addr=172.22.39.3:47466 fd=9 name= age=29 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=938 addr=172.22.39.1:54856 fd=11 name= age=25 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=939 addr=172.22.39.1:54860 fd=12 name= age=25 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=946 addr=172.22.39.2:46864 fd=10 name= age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=940 addr=172.22.39.1:54864 fd=13 name= age=25 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=941 addr=172.22.39.1:54878 fd=14 name= age=25 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=947 addr=172.22.39.4:55964 fd=8 name= age=12 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
127.0.0.1:6379> client list
id=1065 addr=172.22.39.1:46710 fd=17 name= age=30 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=1066 addr=172.22.39.1:46720 fd=18 name= age=30 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=903 addr=127.0.0.1:42950 fd=19 name= age=465 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 qbuf-free=32742 obl=0 oll=0 omem=0 events=r cmd=client
id=1059 addr=172.22.39.1:46648 fd=11 name= age=30 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=1060 addr=172.22.39.1:46656 fd=12 name= age=30 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=1069 addr=172.22.39.3:45942 fd=9 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=1067 addr=172.22.39.2:52166 fd=10 name= age=24 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=1061 addr=172.22.39.1:46660 fd=13 name= age=30 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=1062 addr=172.22.39.1:46674 fd=14 name= age=30 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=1068 addr=172.22.39.4:53398 fd=8 name= age=17 idle=17 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=1063 addr=172.22.39.1:46686 fd=15 name= age=30 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=1064 addr=172.22.39.1:46694 fd=16 name= age=30 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
If you see ids in the output changing when i run client list command after sometime.
I tried to put connection stats logger to see how connection are used. But even when idle count greater than 0, i'm getting JedisExhaustedPoolException exception.
One more detail is even when we restart application issue does not goes away. But when we restart redis it start working again. but problem occur again after sometime.