python 2.7: how to catch keyboard interrupt in program with more than 25 threads

Question

I want to stop my program when the user presses ctrl-C. The following answer suggests catching the KeyboardInterrupt exception.

python: how to terminate a thread when main program ends

Sometimes it works. But in the following example, it stops working after I increase the number of threads from 25 to 30.

import threading, sys, signal, os

stderr_lock = threading.Lock()

def Log(module, msg):
    with stderr_lock:
        sys.stderr.write("%s: %s\n" % (module, msg))

class My_Thread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        Log("Init", "Initing.")
        self.start()
    def run(self):
        try:
            while True:
                Log("Run", "Running.")
        except KeyboardInterrupt:
            os._exit(0)

for i in range(30):
    My_Thread()

# trap ctrl-C in main thread
try:
    while True:
        pass
except KeyboardInterrupt:
    os._exit(0)

This has a very suspiciously similar feel to the following question:

Thread-Safe Signal API in Python 2.7

In that case, I was unable to catch signals after increasing the number of threads beyond 87.

Answer 1

There are actually two different issues with your code that gives this behavior. The first is that your threads should be made into daemon threads, so that they automatically stops when the main thread exits, the second is that your try block does not encapsulate the thread creation and start-up.

When you create a number of threads, the thread creation won't be finished for quite a while (since it is continuously interrupted by the created threads and the GIL prevents them to run in parallel). Therefore, you send your KeyboardInterrupt before being set up to be handled. However, the KeyboardInterrupt will still kill the main thread (with a Traceback), but not the child threads.

Thus, your code works if you modify it as:

import threading, sys, signal, os

stderr_lock = threading.Lock()

def Log(module, msg):
    with stderr_lock:
        sys.stderr.write("%s: %s\n" % (module, msg))

class My_Thread(threading.Thread):
    def __init__(self, value):
        threading.Thread.__init__(self)
        self.value = value
        Log("Init", "Initing %d." % self.value)
        self.daemon = True
        self.start()
    def run(self):
        while True:
            Log("Run", "Running %d." % self.value)

# trap ctrl-C in main thread
try:
    for i in range(1000):
        My_Thread(i)

    while True:
        pass
except KeyboardInterrupt:
    os._exit(0)

Note, that making the threads into daemons is not strictly necessary in the current example, but I would consider that to be good practice for threads that are supposed to end when the main program ends.

Answer 2

You may want to read https://stackoverflow.com/a/35430500/1656850, namely:

There are 3 exit functions, in addition to raising SystemExit.

The underlying one is os._exit, which requires 1 int argument, and exits immediately with no cleanup. It's unlikely you'll ever want to touch this one, but it is there.

sys.exit is defined in sysmodule.c and just runs PyErr_SetObject(PyExc_SystemExit, exit_code);, which is effectively the same as directly raising SystemExit. In fine detail, raising SystemExit is probably faster, since sys.exit requires an LOAD_ATTR and CALL_FUNCTION vs RAISE_VARARGS opcalls. Also, raise SystemExit produces slightly smaller bytecode (4bytes less), (1 byte extra if you use from sys import exit since sys.exit is expected to return None, so includes an extra POP_TOP).

The last exit function is defined in site.py, and aliased to exit or quit in the REPL. It's actually an instance of the Quitter class (so it can have a custom repr, so is probably the slowest running. Also, it closes sys.stdin prior to raising SystemExit, so it's recommended for use only in the REPL.

As for how SystemExit is handled, it eventually causes the VM to call os._exit, but before that, it does some cleanup. It also runs atexit._run_exitfuncs() which runs any callbacks registered via the atexit module. Calling os._exit directly bypasses the atexit step.

so, raise SystemExit may be the preferable way to exit when the exception is caught.