Deciphering large program flow in Python

Question

I'm in the process of learning how a large (356-file), convoluted Python program is set up. Besides manually reading through and parsing the code, are there any good methods for following program flow?

There are two methods which I think would be useful:

1. Something similar to Bash's "set -x"
2. Something that displays which file outputs each line of output

Are there any methods to do the above, or any other ways that you have found useful?

Answer 1

I don't know if this is actually a good idea, but since I actually wrote a hook to display the file and line before each line of output to stdout, I might as well give it to you…

import inspect, sys

class WrapStdout(object):
    _stdout = sys.stdout
    def write(self, buf):
        frame = sys._getframe(1)
        try:
            f = inspect.getsourcefile(frame)
        except TypeError:
            f = 'unknown'
        l = frame.f_lineno
        self._stdout.write('{}:{}:{}'.format(f, l, buf))
    def flush(self):
        self._stdout.flush()

sys.stdout = WrapStdout()

Just save that as a module, and after you import it, every chunk of stdout will be prefixed with file and line number.

Of course this will get pretty ugly if:

• Anyone tries to print partial lines (using stdout.write directly, or print magic comma in 2.x, or end='' in 3.x).
• You mix Unicode and non-Unicode in 2.x.
• Any of the source files have long pathnames.
• etc.

But all the tricky deep-Python-magic bits are there; you can build on top of it pretty easily.

Answer 2

I'd recommend running the program inside an IDE like pydev or pycharm. Being able to stop the program and inspect its state can be very helpful.

Answer 3

You could look for a cross reference program. There is an old program called pyxr that does this. The aim of cross reference is to let you know how classes refer to each other. Some of the IDE's also do this sort of thing.

Answer 4

Could be very tedious, but using a debugger to trace the flow of execution, instruction by instruction could probably help you to some extent.

import pdb
pdb.set_trace()