Unhandled Exception in Flup

Question

I am facing the dreaded "Unhandled Exception" raised by Flup. The sad part is its raised at the webserver (lighttpd+flup) level and not at the application level(Django). So no 500 email is raised about where the problem is.

Our entire team struggled hard to cleanup the codebase, incase of any ambigous imports and someones of that sort, just to be eliminate the chances of raising errors due to the ambiguous imports. And we cleaned up many things in the code. Still the same exception.

To be frank I am really frustrated with Flup's error handling. It doesn't tell you anything. Worst of all, it shows the same "Unhandled Exception" to the Users. How do I get pass this?

I checked the lighttpd logs. All I see is "Interface Error/Connection already closed." Its only occurring when my applicaiton is running in FCGI mode. So the problem is with how flup is actually dealing with my code(application). How do I get pass this?

I checked for alternatives for flup, but Django depends on flup explicitly(which is one more restriction, and puzzled me)(Reference:django_src/django/core/servers/fastcgi.py line:100 / 131)

How do I debug (atleast) this scenario and solve the problem? Please help me out. The application has been down for 3 days.

Answer 1

I don't use lighttpd or flup, so this isn't an answer as much as it is hints.

I'd start by trying to run PDB in your app file, or at least writing to a logfile before you call the flup server .run() method. That way you can identify the problem as being in fastcgi or in flup. See the section called Python Interactive Debugger in the mod_wsgi wiki. That might give you ideas on how to do the same thing with lighttpd and flup.

Then, if the problem is flup you'll catch the exception in pdb and can debug from there.

If the problem is lighttpd then you probably have some kind of problem in your config file, or maybe a problem with the way lighttpd was built. Maybe there's a system library mismatch between lighttp and its fastcgi module?

Try running your app under nginx+fastcgi and see if that works or at least gives you better error messages.

Btw, the author of flup hates FCGI and doesn't even use flup anymore... I'd recommend switching to nginx or apache+mod_wsgi.

Answer 2

This indicates error well before Django starts processing request, like syntax error within settings module. The fastest way to debug such problem is to turn on FastCGI debug. This is the line 128 in django/core/servers/fastcgi.py:

wsgi_opts['debug'] = False # Turn off flup tracebacks

Then run the app with such modified Django, You'll see the Flup traceback in whole its glory.

Answer 3

I hit something similar - we've got Django behind NGINX, and we allow Django to handle 500s - this works 99.9% of the time, but when we're doing upgrades, sometimes these "Unhandled Exceptions" slip through.

Django doesn't override flup's hooks for handling errors, so we'll need to do that ourselves, and let Django handle these errors.

First override flup.server.BaseFCGIServer.error to errors through Django. Then we'll tell Django to use our modified BaseFCGIServer to see those errors.

Since Python is awesome, we'll cheat and just monkeypatch the whole thing in one place, django.core.servers.fastcgi.py. Here we go:

# django.core.servers.fastcgi.py

def runfastcgi(argset=[], **kwargs):
    # ...

    # Paste his hack right after the `module` try/catch.

    # Override BaseFCGIServer.error to use Django error handling.
    # http://trac.saddi.com/flup/browser/flup/server/fcgi_base.py#L1210
    def patch_error(self, req):
        import sys
        from django.conf import settings
        from django.core import urlresolvers
        from django.core.handlers.wsgi import WSGIRequest

        urlconf = settings.ROOT_URLCONF
        urlresolvers.set_urlconf(urlconf)
        resolver = urlresolvers.RegexURLResolver(r'^/', urlconf)

        # No access to 'environ' so rebuild WSGIRequest.
        # http://trac.saddi.com/flup/browser/flup/server/fcgi_base.py#L1077
        environ = req.params
        environ.update(self.environ)
        environ['wsgi.version'] = (1,0)
        environ['wsgi.input'] = req.stdin
        self._sanitizeEnv(environ)        
        wsgireq = WSGIRequest(environ)

        # http://code.djangoproject.com/browser/django/trunk/django/core/handlers/base.py#L177    
        response = self.application.handle_uncaught_exception(wsgireq, resolver, sys.exc_info())

        # TODO: NGINX figures this out, but other servers might not.
        # http://trac.saddi.com/flup/browser/flup/server/fcgi_base.py#L1104
        req.stdout.write('Status: 500\r\n')
        req.stdout.write('Content-Type: text/html\r\n\r\n' + response.content)    

    WSGIServer.error = patch_error

Now you can enjoy Django stack traces even for flup level errors!