Qt: run unit tests from multiple test classes and summarize the output from all of them

Question

Qt comes with QTest, and there are some docs: for example, an official tutorial.

However, QTest encourages you to organize unit tests as separate executables. There is special macro for this, that generates main(): QTEST_MAIN()

To be honest, I really dislike this approach: generally, it is much more useful to run all tests at once, in order to make sure that recent changes haven't broken anything. Sometimes, it is useful to mask out some test or execute some individual test, but this is an exception, not the rule.

So, I want to run all the tests at once. Ok, I can write my own main() that executes all tests I want, say, like this:

int main(int argc, char **argv)
{
   int status = 0;

   //-- run all tests
   {
      TestHTCodecISO14230 tc;
      status |= QTest::qExec(&tc, argc, argv);
   }

   {
      TestHTDataMsg tc;
      status |= QTest::qExec(&tc, argc, argv);
   }

   return status;
}

And it does run all tests, but the problem is that I don't have convenient summary of all tests. Say, for the two tests above, I have two separate summaries:

********* Start testing of TestHTCodecISO14230 *********
Config: Using QtTest library 5.4.1, Qt 5.4.1 (i386-little_endian-ilp32 shared (dynamic) release build; by GCC 4.6.1)
PASS   : TestHTCodecISO14230::initTestCase()
PASS   : TestHTCodecISO14230::decode_summary()
PASS   : TestHTCodecISO14230::encode()
PASS   : TestHTCodecISO14230::decode_encoded()
PASS   : TestHTCodecISO14230::cleanupTestCase()
Totals: 5 passed, 0 failed, 0 skipped, 0 blacklisted
********* Finished testing of TestHTCodecISO14230 *********
********* Start testing of TestHTDataMsg *********
Config: Using QtTest library 5.4.1, Qt 5.4.1 (i386-little_endian-ilp32 shared (dynamic) release build; by GCC 4.6.1)
PASS   : TestHTDataMsg::initTestCase()
PASS   : TestHTDataMsg::test1()
PASS   : TestHTDataMsg::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted
********* Finished testing of TestHTDataMsg *********

The fact that returned status will be non-zero in case of error is surely helpful, but is would be much more helpful if I have summary as well:

Totals: 8 passed, 0 failed, 0 skipped, 0 blacklisted

From what I see, it is impossible: I can't find the way to programmatically get number of passed, failed, skipped and blacklisted tests: qExec() is just a function in the QTest namespace, so, it is impossible to gather some additional info after it executes.

Well, it is possible to parse the output string, but, ugh...

To me, it looks like poor design. It would be probably much better to make QTest as a class, then make instance of it and feed some test classes to it. Then, some additional information could be gathered from an instance.

Or, maybe I've missed something.

So, the question is: is it possible with QTest to have summary output of all unit test classes?

Answer 1

As I wrote in my comment, I would construct my test classes in the following way:

class MyTests: public QObject
{
    Q_OBJECT
public:
    MyTests() : m_executed(0), m_failed(0)
private slots:
    [..]
    // This function will be called after each test
    void cleanup()
    {
        m_executed++;
        if (currentTestFailed()) {
            m_failed++;
        }        
    }

    // Output the summary of the test execution.
    void report() const
    {
        qDebug() << "Totals:"
                 << m_executed - m_failed  << "passed,"
                 << m_failed << "failed";
    }
private:
    int m_executed;
    int m_failed;
};

If you have multiple instances of MyTests class, you can extend its API and sum up the execution results producing the global test execution report. Just use the whole strength of C++ classes.