Running flask server, nosetests and coverage

Question

I am writing an api in Flask. I have couple of views that return json responses and I have written some unit tests that will check if those views are working properly and are returning correct data. Then I turned on coverage plugin for nosetests (in my case nose-cov).

And here's where my problem starts, coverage is not seeing my views as being executed by tests.

First some base code to give you full picture:

My view:

def get_user(uid):
    """Retrieve user.

    Args:
        uid (url): valid uid value

    Usage: ::

        GET user/<uid>/

    Returns:
      obj:
       ::

        {
            'data': {
                `response.User`,
            },
            'success': True,
            'status': 'get'
        }
    """
    if not uid:
        raise exception.ValueError("Uid is empty")

    obj = db_layer.user.get_user(uid=value)

    return {
        'data': {
            obj.to_dict(),  # to_dict is helper method that converts part of orm into dict
        },
        'success': True,
        'status': 'get'
    }

My test:

class TestUserViews(base.TestViewsBase):
    def test_get_user(self):
        uid = 'some_uid_from_fixtures'
        name = 'some_name_from_fixtures'

        response = self.get(u'user/uid/{}/'.format(uid))
        self.assertEqual(response.status_code, 200)

        user_data = json.loads(response.text)['data']
        self.assertEqual(name, user_data['username'])
        self.assertEqual(uid, user_data['uid'])

    def get(self, method, headers=None):
        """
        Wrapper around requests.get, reassures that authentication is
        sorted for us.
        """
        kwargs = {
            'headers': self._build_headers(headers),
        }

        return requests.get(self.get_url(method), **kwargs)

    def get_url(self, method):
        return '{}/{}/{}'.format(self.domain, self.version, method)

    def _build_headers(self, headers=None):
        if headers is None:
            headers = {}

        headers.update({
            'X-Auth-Token': 'some-token',
            'X-Auth-Token-Test-User-Id': 'some-uid',
        })

        return headers

To run test suite I have special shell script that performs few actions for me:

#!/usr/bin/env bash

HOST="0.0.0.0"
PORT="5001"

ENVS="PYTHONPATH=$PYTHONPATH:$PWD"

# start server 
START_SERVER="$ENVS python $PWD/server.py --port=$PORT --host=$HOST"

eval "$START_SERVER&"

PID=$!

eval "$ENVS nosetests -s --nologcapture --cov-report html --with-cov"

kill -9 $PID

After that view is reported as not being executed.

Answer 1

Ok guys, 12h later I found a solution. I have checked flask, werkzeug, requests, subprocess and thread lib. To only learn that problem is somewhere else. Solution was easy in fact. The bit of code that has to modified is execution of server.py. We need to cover it as well and then merge results generated by server.py and generated by nosetests. Modified test-runner.sh looks as follows:

#!/usr/bin/env bash

HOST="0.0.0.0"
PORT="5001"

ENVS="COVERAGE_PROCESS_START=$PWD/.apirc PYTHONPATH=$PYTHONPATH:$PWD"

START_SERVER="$ENVS coverage run --rcfile=.apirc $PWD/server.py --port=$PORT --host=$HOST"

eval "$START_SERVER&"

eval "$ENVS nosetests -s --nologcapture --cov-config=.apirc --cov-report html --with-cov"

# this is important bit, we have to stop flask server gracefully otherwise
# coverage won't get a chance to collect and save all results
eval "curl -X POST http://$HOST:$PORT/0.0/shutdown/"

# this will merge results from both coverage runs
coverage combine  --rcfile=.apirc
coverage html --rcfile=.apirc

Where .apirc in my case is looks as follows:

[run]
branch = True
parallel = True
source = files_to_cover/

[html]
directory = cover

Last thing we need to do is to build into our flask, view that will allow us to greacefully shut down the server. Previously I was brute killing it with kill -9 and it used to kill not only server but coverage as well.

Follow this snippet: http://flask.pocoo.org/snippets/67/

And my view is like that:

def shutdown():
    if config.SHUTDOWN_ALLOWED:
        func = request.environ.get('werkzeug.server.shutdown')
        if func is None:
            raise RuntimeError('Not running with the Werkzeug Server')
        func()

        return 'Server shutting down...'

It is important to use nose-cov instead of standard coverage plugin, as it uses rcfile and allows to configure more. In our case parallel is the key, please note data_files variable is not working for nose-coverage, so you cannot override it in .apirc and you have to use default values.

After all of that, your coverage will be all but shining with valid values.

I hope it going to be helpful for someone out there.