PhpUnit + Symfony: Why coverage shows white instead of red and gives 100% on untested class?

Question

I have a problem and I have created an empty project to reproduce the minimal situation that makes it repeatable.

Problem

A project with untested classes gives 100% coverage. The problematic methods are not indirectly called from somewhere else. Although there are other methods of the untested class that are indirectly called when testing another class.

How to reproduce

Step 1: Create an empty new symfony project.

• I have created a new symfony 3.3 project with this command: symfony new coverage_trial_to_be_deleted

Result: If I run vendor/bin/simple-phpunit --coverage-html coverageReport I get a fully tested project, as expected, as the sample contains one default test for one default controller.

Step 2: Delete the controller and create two commands, but do not cover them.

• I then eliminate the controller by removing the full src/Controller directory, the test for the controller, and the references to that directroy in the app configuration.
• I then create a src/Command directory with 2 commands: DummyACommand.php and DummyBCommand.php.
• I then create a stupid test that performs an assertTrue( true ); to have something to report, but does not invoke the commands at all.

Result: This works correct. It then reports 0% on the Command directory, like this:

In particular, inside the Command I can see the 2 commands at 0%, which is normal:

And finally if I enter the second of the commands, the "B" command, say DummyBCommand, I can still see that neither the configure() nor the execute() methods have been covered, and I see, as expected, a red area on each of those methods:

Up to here, everything works as expected.

Step 3: Add test for command A only and do NOT test command B.

I then add a new test, named DummyACommandTest.php with the following content:

<?php

declare( strict_types=1 );

namespace Tests\AppBundle\Command;

use AppBundle\Command\DummyACommand;
use Symfony\Component\Console\Tester\CommandTester;
use Symfony\Bundle\FrameworkBundle\Console\Application;
use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase;

class EnvironmentListCommandTest extends KernelTestCase
{
    public function testExecute()
    {
        $kernel = $this->createKernel();
        $kernel->boot();

        $application = new Application( $kernel );
        $application->add( new DummyACommand() );

        $command = $application->find( 'dummy:a' );
        $commandTester = new CommandTester( $command );
        $commandTester->execute(
            [
                'command' => $command->getName(),
            ]
        );

        $output = $commandTester->getDisplay();
        $this->assertContains( 'dummy A command', $output );
    }
}

The test gives green light, as expected, as it invokes the DummyACommand and executes it, then it checks for the output and it really contains what it is expected:

It says 2 tests because the assertTrue( true ); is still there.

Result: Now, if I look at the coverage of DummyBCommand I should see the configure() method covered, because the fact of testing A invokes a $application->find( 'dummy:a' ); and I guess it explores all the commands.

I really don't understand this very much, as I only do the $application->add( new DummyACommand() ); and I don't know when dummy:b is loaded, but anyway let's give this hypothesis a chance.

So I don't mind seeing the configure() in green (Annotation [1] in the following image).

But what I don't like is that then, the execute() of B instead of appearing in red (noone calls that execution, should appear exactly as in step 2), it appears in white!!! :| like in annotation [2] here:

So for some reason... testing Command A makes the PHPUnit to change its consideration of what is "potentially executable" in Command B and I don't understand why.

This makes the full project to be reported as 100% tested, while it is false, like in here:

but of course this information does not reflect the reality: the execute() method of the command B is not covered and never executed.

I'd hope that the report tells me 75% line coverage (3 of 4) and 50% class coverage (1 of 2).

So... Question:

Why does PhpUnit change its consideration about what is potentially executable code in class B while adding a test for class A?

How can I instruct PhpUnit to see B as uncovered?

Thanks! Xavi.

Answer 1

Found!!

Quick solution

Update the xdebug library here: https://xdebug.org/wizard.php

Detailed solution

As you can see in the question's images, I was using the Xdebug 2.4.0 because that's the one that just was in my system, a stable Ubuntu 16.04.

As per this thread: https://github.com/sebastianbergmann/php-code-coverage/issues/411 during one full year those folks (thanks you all) they were pursuing the same problem: Lines not being reported as executable.

Finally they came to the conclusion that there was a bug in XDebug and it finally was reported and corrected.

This is: xdebug was improerply reporting data to phpunit, so phpunit was running on erroneous hypothesis.

I upgraded here https://xdebug.org/wizard.php (the link I found in that thread), and I compiled a new xdebug as instructed.

Here you can see that the project is not now 100%, but 75% in lines and 50% in classes (as expected):

Here you can see that the Command A reports 100% (which is okey, because it is the one I test) and Command B is reported 50% in lines and 0% in class, as expected, as I did not write a test for it. The method configure() is run on kernel boot and application loading:

And if we enter in Command B (ie: DummyBCommand) we can finally see that the method execute() is marked as red, not white:

In this image you can see in [1] that the configure() is 100% and the execute() is 0%, which is correct.

In [2] you can see the colors. execute() is marked red, as expected.

And finally, in [3] you can see that the XDebug version is the 2.5.4, the latest stable on 10/sep/2017 (or at least, the one that the update wizard suggested me to install).

So... download, configure, compile, install and boom, done!