python run function in external module containing doctest.testmod()

Question

I am trying to automate testing across several modules. All of these modules have a "test()" function with their unit-tests. Some modules are basic and their tests contain simple statements, but most modules have unittest or doctest. I'm having the most trouble dynamically importing and running doctest.

For example, here is a module sample.py

class sample:
    """
    >>> import sample
    >>> print sample.hello()
    Hello
    """
    def hello():
        return "Hello"

def test():
    import doctest
    doctest.testmod(name=__name__, verbose=True)

And here is my file run_all_tests.py:

# assume I already have a list of all my files to test
for file in all_my_files:
    temp_module = __import__(file)
    temp_module.test()

This doesn't work and I always get this error:

1 items had no tests:
    sample
0 tests in 1 items.
0 passed and 0 failed.
Test passed.

Please help me understand the problem.

Would Nose be a good alternative? I don't want to use it because I won't know beforehand if a module uses doctests, unittests or simple statements. But do let me know if that's not true/you have another alternative entirely!

Answer 1

Use doctest.DocTestSuite. It takes a module, extracts all doctests that exist there, and returns it as a unittest.TestSuite. Then, running the tests is a piece of pie. Your code would look like this:

for f in all_my_files:
    temp_module = __import__(f)
    test_suite = doctest.DocTestSuite(temp_module)
    unittest.TextTestRunner().run(test_suite)

Why your code wasn't working

From doctest's testmod documentation:

Test examples in docstrings in functions and classes reachable from module m (or module __main__ if m is not supplied or is None), starting with m.__doc__.

So, since you left out the first argument (m), the module __main__ is passed to testmod. So the doctests that were run were the doctests in the module that contain the for loop. You can see this for yourself:

run_tests.py

"""
    >>> print 'oops'
    oops
"""
# assume I already have a list of all my files to test
for file in all_my_files:
    temp_module = __import__(file)
    temp_module.test()

If you run your example now (before fixing it) you'll see that you'll get:

Trying:
    print 'oops'
Expecting:
    oops
ok
1 items passed all tests:
   1 tests in sample
1 tests in 1 items.
1 passed and 0 failed.
Test passed.

Clearly showing that the doctests running are those in run_tests.py. The name argument only changes the name that appears in the message (sample).