Go (Redis with Redigo) suddenly drops connections in a pubsub loop

Question

My open-source project - Ballot - has been running on AWS against an EC2/Redis instance for years without a single problem.

I recently moved it to DigitalOcean, and PubSub started having problems with dropped connections.

My Receive loop is pretty straightforward, but now in a few minutes (or hours) I get an EOF.

I upgraded from Go 1.13 to 1.21 and got the latest versions of libraries, but the problem just manifested itself as a more verbose error about a connection having gone away.

I cranked down IdleTimeout here from 240 to 30 seconds, and it only sometimes helps. Eventually the Go routine will panic and exit.

It is somehow related to the new host and new Redis, v4 to Digital Ocean's Redis 7, but I cannot find how.

Answer 1

Turns out advice in another question helped out here: How to write better Receive() in Golang for Redis(redigo) Pubsub?

It's easy to find examples of less resilient connection loops - and mine just happened to work well for a long time in the right context.

Basically - two loops: the outer loop being the "guard" for the current connection.