TechQA.

ActionCable disconnects from redis server after 5 minutes

1.2k views Asked by At

I have a Rails API that has Redis as an adapter of ActionCable.

I only use ActionCable for a simple chat as a part of the whole system.

Everything works perfect, I can subscribe, send messages, receive messages. The problem comes after 5 minutes the server has started, I get an ECONNRESET error in redis gem: Connection lost (ECONNRESET) (Redis::ConnectionError)

It's been 3 days since i've started trying to solve this issue, i've tried changing redis-server configuration, i've tried changing the way redis initializes on the rails app, i've tried changing the cable.yml configuration, i've tried changing actioncable config and nothing seems to solve this issue, i'm running out of ideas.

I also can't rescue this exception, it just breaks the whole rails app, thats another thing I couldn't solve that's so frustrating.

Btw I am testing in development environment.

This is my cable config:

production:
  adapter: redis
  url: redis://192.168.1.6:6379

development:
  adapter: redis
  url: redis://192.168.1.5:6379
  channel_prefix: ChatChannel
  timeout: 30
  max_conns: 8000
  max_persistent_conns: 8000
  
test:
  adapter: async

This is my redis initializer:

require 'redis'

## Added rescue condition if Redis connection is failed
begin
    puts "CREATING NEW INSTANCE OF REDIS"
    $redis = Redis.new(
        :host => "192.168.1.5", 
        :port => 6379, 
        :reconnect_attempts => 10,
        :reconnect_delay => 1.5,
        :reconnect_delay_max => 10.0
    )
    $redis.ping
rescue Errno::ECONNRESET => e
    puts "THERE WAS AN ERROR"
    puts e
end

This is the output of the rails app:

web_1  | Exiting
web_1  | #<Thread:0x0000559ee244b1d8 /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:151 run> terminated with exception (report_on_exception is true):
web_1  | /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:275:in `rescue in io': Connection lost (ECONNRESET) (Redis::ConnectionError)
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:267:in `io'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:279:in `read'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:147:in `block (3 levels) in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:146:in `loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:146:in `block (2 levels) in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:248:in `block (2 levels) in process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:389:in `ensure_connected'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:238:in `block in process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:325:in `logging'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:237:in `process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:145:in `block in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:300:in `with_socket_timeout'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:144:in `call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/subscribe.rb:44:in `subscription'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/subscribe.rb:14:in `subscribe'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:3507:in `_subscription'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:2326:in `block in subscribe'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:2325:in `subscribe'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:84:in `block in listen'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:313:in `with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:75:in `block in with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:74:in `with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:81:in `without_reconnect'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:81:in `listen'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:155:in `block in ensure_listener_running'
web_1  | /usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb:63:in `block in _read_from_socket': Connection reset by peer (Errno::ECONNRESET)
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `_read_from_socket'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb:45:in `gets'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb:378:in `read'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:280:in `block in read'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:268:in `io'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:279:in `read'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:147:in `block (3 levels) in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:146:in `loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:146:in `block (2 levels) in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:248:in `block (2 levels) in process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:389:in `ensure_connected'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:238:in `block in process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:325:in `logging'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:237:in `process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:145:in `block in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:300:in `with_socket_timeout'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:144:in `call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/subscribe.rb:44:in `subscription'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/subscribe.rb:14:in `subscribe'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:3507:in `_subscription'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:2326:in `block in subscribe'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:2325:in `subscribe'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:84:in `block in listen'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:313:in `with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:75:in `block in with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:74:in `with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:81:in `without_reconnect'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:81:in `listen'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:155:in `block in ensure_listener_running'
web_1  | /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:275:in `rescue in io': Connection lost (ECONNRESET) (Redis::ConnectionError)
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:267:in `io'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:279:in `read'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:147:in `block (3 levels) in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:146:in `loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:146:in `block (2 levels) in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:248:in `block (2 levels) in process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:389:in `ensure_connected'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:238:in `block in process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:325:in `logging'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:237:in `process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:145:in `block in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:300:in `with_socket_timeout'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:144:in `call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/subscribe.rb:44:in `subscription'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/subscribe.rb:14:in `subscribe'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:3507:in `_subscription'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:2326:in `block in subscribe'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:2325:in `subscribe'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:84:in `block in listen'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:313:in `with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:75:in `block in with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:74:in `with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:81:in `without_reconnect'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:81:in `listen'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:155:in `block in ensure_listener_running'
web_1  | /usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb:63:in `block in _read_from_socket': Connection reset by peer (Errno::ECONNRESET)
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb:52:in `_read_from_socket'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb:45:in `gets'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/connection/ruby.rb:378:in `read'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:280:in `block in read'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:268:in `io'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:279:in `read'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:147:in `block (3 levels) in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:146:in `loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:146:in `block (2 levels) in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:248:in `block (2 levels) in process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:389:in `ensure_connected'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:238:in `block in process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:325:in `logging'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:237:in `process'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:145:in `block in call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:300:in `with_socket_timeout'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:144:in `call_loop'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/subscribe.rb:44:in `subscription'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/subscribe.rb:14:in `subscribe'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:3507:in `_subscription'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:2326:in `block in subscribe'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:2325:in `subscribe'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:84:in `block in listen'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis/client.rb:313:in `with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:75:in `block in with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
web_1  |    from /usr/local/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:74:in `with_reconnect'
web_1  |    from /usr/local/bundle/gems/redis-4.2.5/lib/redis.rb:81:in `without_reconnect'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:81:in `listen'
web_1  |    from /usr/local/bundle/gems/actioncable-6.0.3.1/lib/action_cable/subscription_adapter/redis.rb:155:in `block in ensure_listener_running'
nulimit-web_web_1 exited with code 1

This is the output of the redis-server:

1:M 08 Dec 2020 20:03:00.438 - Client closed connection
ruby redis actioncable rails-api redis-server
Original Q&A
0

There are 0 answers

Related Questions in RUBY

Related Questions in REDIS

Related Questions in ACTIONCABLE

Related Questions in RAILS-API

Related Questions in REDIS-SERVER

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Trending Questions