Paramiko inside Python Daemon causes IOError

Question

I'm trying to execute ssh commands using paramiko from inside a python daemon process. I'm using the following implementation for the daemon: https://pypi.python.org/pypi/python-daemon/

When the program is started pycrypto raises an IOError with a Bad file descriptor when paramiko tries to connect. If I remove the daemon code (just uncomment the last line and comment the two above) the ssh connection is established as expected.

The code for a short test program looks like this:

#!/usr/bin/env python2
from daemon import runner
import paramiko

class App():

    def __init__(self):
        self.stdin_path = '/dev/null'
        self.stdout_path = '/dev/tty'
        self.stderr_path = '/dev/tty'
        self.pidfile_path =  '/tmp/testdaemon.pid'
        self.pidfile_timeout = 5

    def run(self):
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.load_system_host_keys()
        ssh.connect("hostname", username="username")
        ssh.close()

app = App()
daemon_runner = runner.DaemonRunner(app)
daemon_runner.do_action()
#app.run()

The trace looks like this:

Traceback (most recent call last):
  File "./daemon-test.py", line 31, in <module>
    daemon_runner.do_action()
  File "/usr/lib/python2.7/site-packages/daemon/runner.py", line 189, in do_action
    func(self)
  File "/usr/lib/python2.7/site-packages/daemon/runner.py", line 134, in _start
    self.app.run()
  File "./daemon-test.py", line 22, in run
    ssh.connect("hostname", username="username")
  File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 311, in connect
    t.start_client()
  File "/usr/lib/python2.7/site-packages/paramiko/transport.py", line 460, in start_client
    Random.atfork()
  File "/usr/lib/python2.7/site-packages/Crypto/Random/__init__.py", line 37, in atfork
_UserFriendlyRNG.reinit()
  File "/usr/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 224, in reinit
_get_singleton().reinit()
  File "/usr/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 171, in reinit
    return _UserFriendlyRNG.reinit(self)
  File "/usr/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 99, in reinit
    self._ec.reinit()
  File "/usr/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 62, in reinit
    block = self._osrng.read(32*32)
  File "/usr/lib/python2.7/site-packages/Crypto/Random/OSRNG/rng_base.py", line 76, in read
data = self._read(N)
  File "/usr/lib/python2.7/site-packages/Crypto/Random/OSRNG/posix.py", line 65, in _read
    d = self.__file.read(N - len(data))
IOError: [Errno 9] Bad file descriptor

I'm guessing this has something to do with the stream redirection when the daemon spawns. I've tried to set them all to /dev/tty or even to a normal file but nothing works.

When I run the program with strace I can see that something tries to close a file twice and that's when I get the error. But I couldn't find out which file the descriptor actually points to (strace shows a memory location that doesn't seem to be set anywhere).

Answer 1

This is a known issue that I am actually experiencing myself (which is what led me to this question). Basically, it has to do with the definition of a UNIX daemon process and the way paramiko implements its random number generator (RNG).

If you refer to PEP 3143 - Standard daemon process library, the first step in becoming a correct daemon is to "close all open file descriptors." Unfortunately, this closes the file descriptor to /dev/urandom which is used in the Crypto module's RNG which is in turn used by paramiko.

There are some workarounds for the moment, but the author has indicated that he doesn't currently have time to pursue this bug (although the last post in the first link is by the author and is 8 days old as of this writing).

• Daemonizing after importing paramiko breaks the random number generator
• EAGAIN on file when using RNG after daemon fork

In summary, if you import paramiko after your process becomes a daemon, then it should work as desired because the file descriptor will have been opened after the daemonizing closes all file descriptors.

The user @xraj also had a hackish, yet clever workaround for finding and preserving the file descriptor to /dev/urandom when daemonizing (first link above):

import os
from resource import getrlimit, RLIMIT_NOFILE

def files_preserve_by_path(*paths):
    wanted=[]
    for path in paths:
        fd = os.open(path, os.O_RDONLY)
        try:
            wanted.append(os.fstat(fd)[1:3])
        finally:
            os.close(fd)

    def fd_wanted(fd):
        try:
            return os.fstat(fd)[1:3] in wanted
        except OSError:
            return False

    fd_max = getrlimit(RLIMIT_NOFILE)[1]
    return [ fd for fd in xrange(fd_max) if fd_wanted(fd) ]

daemon_context.files_preserve = files_preserve_by_path('/dev/urandom')

Answer 2

This recently happens for daemons and multithreading applications which makes mass close() in cycle of separated thread. I've found the problem in class pipe.PosixPipe. There is no synchronization between set() and close() methods. Methods of PosixPipe could read/write and close descriptor of socket at the same time. Issue was created: https://github.com/paramiko/paramiko/issues/692 Pull was requested: https://github.com/paramiko/paramiko/pull/691/files