Terminate a hung redis pubsub.listen() thread

Question

Related to this question I have the following code which subscribes to a redis pubsub queue and uses the handler provided in __init__ to feed the messages to the class that processes them:

from threading import Thread
import msgpack

class Subscriber(Thread):

    def __init__(self, redis_connection, channel_name, handler):
        super(Subscriber, self).__init__(name="Receiver")
        self.connection = redis_connection
        self.pubsub = self.connection.pubsub()
        self.channel_name = channel_name
        self.handler = handler
        self.should_die = False

    def start(self):
        self.pubsub.subscribe(self.channel_name)
        super(Subscriber, self).start()

    def run(self):
        for msg in  self.pubsub.listen():
            if self.should_die:
                return
            try:
                data = msg["data"]
                unpacked = msgpack.unpackb(data)
            except TypeError:
                # stop non-msgpacked, invalid, messages breaking stuff
                # other validation happens in handler
                continue
            self.handler(unpacked)

    def die(self):
        self.should_die = True

In the linked question above, it is noted that pubsub.listen() never returns if the connection is dropped. Therefore, my die() function, while it can be called, will never actually cause the thread to terminate because it is hanging on the call to listen() inside the thread's run().

The accepted answer on the linked question mentions hacking redis-py's connection pool. I really don't want to do this and have a forked version of redis-py (at least until the fix is hopefully accepted into master), but I've had a look at the redis-py code anyway and don't immediately see where this change would be made.

Does anyone have an idea how to cleanly solve the hanging redis-py listen() call?

What issues will I incur by directly using Thread._Thread__stop?

Answer 1

To close this out after so many years. It ended up being a bug in the redis library. I debugged it and submitted the PR. It shouldn't occur any more.