using assertRaises - handling propagated exceptions

Question

I have some code where I'm testing for a wrapped exception, when it failed and the exception propagated I thought the error message and back trace wasn't verbose enough, primarily because it didn't tell me what was expected vs. the test, I would like details of the exception and the expectation.

I adjusted my test (see example code below). I would like to know if this type of approach is valid and if any of the Python testing or mocking frameworks allow to implement it directly? (currently I'm using unittest and mox)

One of the answers to this question briefly touches on the appropriateness of using self.fail in this scenario, but doesn't really elaborate. My assumption is that if I try to limit the test to one area I'm okay to fail the test.

Note: The code example should fail if you run it, to demonstrate the behaviour I would like to see. I'm using Python 2.7, Mox 0.5.3

import sys
import urllib2
from contextlib import closing

try:
    import lxml.etree as ET
except ImportError:
    import xml.etree.ElementTree as ET


class Defect(Exception):
    """Wrapped exception, for module error detection"""
    def __init__(self, *args):
        Exception.__init__(self, *args)
        self.wrapped_exc = sys.exc_info()


class StudioResources:
    """Dummy class"""
    def _opener(self, request, html=False):
        with closing(urllib2.urlopen(request)) as response:
            try:
                if html:
                    import lxml.html
                    return lxml.html.parse(response)
                else:
                    return ET.parse(response)
            except urllib2.HTTPError, e:
                if e.code in [400, 500]: # Bad Request, Internal Server Error
                    raise Defect, "report error to the library maintainer"
                else:
                    raise


###
# Tests
###
import unittest
import mox
import traceback
import difflib
import urllib
import httplib


def format_expectation(exc_expected=None, exc_instance=None):
    """Synopsis - For exceptions, inspired by _AssertRaisesContext

    try:
        self.assertRaises(myexc, self.studio._opener, None)
    except Exception, e:
        self.fail(format_expectation(exc_expected=myexc, exc_instance=e))
    """
    if not isinstance(exc_expected, type) or exc_instance is None:
        raise ValueError, "check __init__ args"

    differ = difflib.Differ()
    inst_class = exc_instance.__class__
    def fullname(c): return "%s.%s" % (c.__module__, c.__name__)
    diff = differ.compare(
        (fullname(inst_class),), (fullname(exc_expected),))
    _str = ("Unexpected Exception type.  unexpected:-  expected:+\n%s"
        % ("\n".join(diff),))
    return _str


class StudioTest(mox.MoxTestBase):
    def setUp(self):
        mox.MoxTestBase.setUp(self)
        self.studio = StudioResources()

    def test_opener_defect(self):
        f = urllib.addinfourl(urllib2.StringIO('dummy'), None, None)
        RESP_CODE = 501
        self.mox.StubOutWithMock(f, 'read')
        self.mox.StubOutWithMock(urllib2, 'urlopen')
        urllib2.urlopen(mox.IgnoreArg()).AndReturn(f)
        f.read(mox.IgnoreArg()).AndRaise(urllib2.HTTPError(
            'http://c.com', RESP_CODE, httplib.responses[RESP_CODE], "", None))
        self.mox.ReplayAll()
        try:
            with self.assertRaises(Defect) as exc_info:
                self.studio._opener(None)
        except Exception, e:
            traceback.print_exc()
            self.fail(format_expectation(exc_expected=Defect, exc_instance=e))
        # check the response code
        exc, inst, tb = exc_info.exception.wrapped_exc
        self.assertEquals(inst.code, RESP_CODE)
        self.mox.VerifyAll()


if __name__ == '__main__':
    unittest.main()

Answer 1

When writing unit tests its always a good idea to limit the test to one thing. I don't see anything wrong with your code, but I would wrap the whole lot in a context manager. I use nose rather than unittest, which treats any AssertionError as a fail (this means that there is no need to call self.fail()), and I've written my own context manager to handle this case. Here is the code if you're interested:

class assert_raises:

    def __init__(self, exception):
        self.exception = exception

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        assert exc_type is self.exception, "Got '{}', expected '{}'"\
            .format('None' if exc_type is None else exc_type.__name__,
                    self.exception.__name__)
        return True

And then use it as in these example:

>>> with assert_raised(ValueError):
...    raise ValueError

>>> with assert_raised(ValueError):
...    pass
Traceback (most recent call last):
    ...
AssertionError: Got 'None', expected 'ValueError'

>>> with assert_raised(ValueError):
...     raise TypeError
Traceback (most recent call last):
    ...
AssertionError: Got 'TypeError', expected 'ValueError'

Since an AssertionError is raised, nose sees it as a failure and prints the full traceback anyway. This is designed for nose, but it would be a trivial matter to adjust it for unittest and mox instead. If you're not too concerned about the exact mode of failure you could even use it as is.