Python tornado OS Error 24 Too many open files

Question

I am using a websocket server coded in Python 3.5. This is needed to provide my website realtime information.

The "server"-library I use is called tornado (version 4.3) which handles the websocket connection and http requests. For backwards compatibility (if a browser does not support websocket yet) I use the python library sockjs-tornado (version 1.0.3) which also adds some additional features to the websocket connection.

This connection uses the protocol wss:// so it is encrypted and loads a certificate for the socket connections. The Problem is that I get an error when the server has been running for quite a while.

ERROR:tornado.application:Exception in callback (<socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('***', 8443)>, <function wrap.<locals>.null_wrapper at 0x6fbeea4769d8>)
Traceback (most recent call last):
  File "/home/website/python/tornado/ioloop.py", line 883, in start
    handler_func(fd_obj, events)
  File "/home/website/python/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/website/python/tornado/netutil.py", line 274, in accept_handler
    callback(connection, address)
  File "/home/website/python/tornado/tcpserver.py", line 239, in _handle_connection
    do_handshake_on_connect=False)
  File "/home/website/python/tornado/netutil.py", line 510, in ssl_wrap_socket
    context = ssl_options_to_context(ssl_options)
  File "/home/website/python/tornado/netutil.py", line 487, in ssl_options_to_context
    context.load_cert_chain(ssl_options['certfile'], ssl_options.get('keyfile', None))
OSError: [Errno 24] Too many open files

I already increased the limit of open files on the linux server, but if the python script was not restarted since 24h then these errors will pop up if about 300 clients are connected. If I restart the script during this time everything will work fine again, even after everyone reconnected.

I do not really know what to do because it seems like that tornado is not closing the certificate files correctly and loading them again on every new connection.

Answer 1

It may not be the case that tornado is leaking file handles; it's just tornado that is hitting the limit after a while because it regularly opens files.

Get the PID for the process in question and look at /proc/[PID]/fd. It will list all the file handles that are currently open. If your code keeps files open due to some reason, they will pile up there.