'No tests were found' error when trying to use Suites API with JUnit5 + Cucumber combination

Question

It seems like no matter what combination of tags and paths I put along with Suites tag, the tests are not discoverable. The same happens if I don't use Suites tag and just run mvn test.

The only way that it works if I explicitly state the location of features inside junit-platform.properties file. That way the features are working just fine if I run mvn test. In case I use the exact same paths in Suite property tags, it doesn't.

Here are the screenshot of configuration(note that junit properties file is empty):

(https://i.stack.imgur.com/GVINK.png)

Some more details:

This is a content of a feature file

Scenario Outline: Verify user is unable to login with invalid credentials
    Given I am on Login page
    When I enter username "<username>" and password "<password>"
    Then I check login is not successful
    Examples:
      | username      | password  |
      | not_existing  | 123456789 |
      | invalid_login | test1234  |

This is how my POM file looks like

<properties>
    <log4j-core.version>2.20.0</log4j-core.version>
    <testng.version>6.14.3</testng.version>
    <owner.version>1.0.12</owner.version>
    <client-java.version>5.1.16</client-java.version>
    <playwright.version>1.35.0</playwright.version>
    <lombok.version>1.18.26</lombok.version>
    <commons-csv.version>1.10.0</commons-csv.version>
    <maven-compiler-plugin.source>1.8</maven-compiler-plugin.source>
    <maven-compiler-plugin.target>1.8</maven-compiler-plugin.target>
    <junit-jupiter-api.version>5.9.3</junit-jupiter-api.version>
    <cucumber-java.version>7.13.0</cucumber-java.version>
    <maven-surefire-plugin.version>3.0.0-M6</maven-surefire-plugin.version>
</properties>


<dependencies>
    <dependency>
        <artifactId>log4j-core</artifactId>
        <groupId>org.apache.logging.log4j</groupId>
        <version>${log4j-core.version}</version>
    </dependency>
    <dependency>
        <groupId>org.testng</groupId>
        <artifactId>testng</artifactId>
        <version>${testng.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.aeonbits.owner</groupId>
        <artifactId>owner</artifactId>
        <version>${owner.version}</version>
    </dependency>
    <dependency>
        <groupId>com.epam.reportportal</groupId>
        <artifactId>client-java</artifactId>
        <version>${client-java.version}</version>
    </dependency>
    <dependency>
        <groupId>org.testng</groupId>
        <artifactId>testng</artifactId>
        <version>${testng.version}</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>com.microsoft.playwright</groupId>
        <artifactId>playwright</artifactId>
        <version>${playwright.version}</version>
    </dependency>
    <dependency>
        <artifactId>lombok</artifactId>
        <groupId>org.projectlombok</groupId>
        <version>${lombok.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-csv</artifactId>
        <version>${commons-csv.version}</version>
    </dependency>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-api</artifactId>
        <version>${junit-jupiter-api.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-params</artifactId>
        <version>${junit-jupiter-api.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-java</artifactId>
        <version>${cucumber-java.version}</version>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-picocontainer</artifactId>
        <version>${cucumber-java.version}</version>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-junit-platform-engine</artifactId>
        <version>${cucumber-java.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.junit.platform</groupId>
        <artifactId>junit-platform-suite-api</artifactId>
        <version>1.10.0-M1</version>
        <scope>test</scope>
    </dependency>
</dependencies>

Answer 1

You've got quite a few problems going on.

1. You did not include the junit-platform-suite implementation as a dependency. Rather you included the junit-platform-suite-api which only contains the annotations not the actual test engine that executes the tests. Read the JUnit 5 documentation for a conceptual understanding.

2. Your JUnit dependencies don't converge. You are mixing v5.9.3 and v1.10.0-M1. Use the JUnit BOM to solve this. You can then declare your JUnit dependencies without version numbers.

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.junit</groupId>
                <artifactId>junit-bom</artifactId>
                <version>5.9.3</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>io.cucumber</groupId>
                <artifactId>cucumber-bom</artifactId>
                <version>7.13.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-java</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-junit-platform-engine</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-suite</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

3. Your runner class CucumberRunner.java does not end with *Test.java and because of that won't be picked up by Surefire. Rename it to CucumberTest.java

https://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html

By default, the Surefire Plugin will automatically include all test classes with the following wildcard patterns:

"**/Test*.java" - includes all of its subdirectories and all Java filenames that start with "Test".

"**/*Test.java" - includes all of its subdirectories and all Java filenames that end with "Test".

"**/*Tests.java" - includes all of its subdirectories and all Java filenames that end with "Tests".

"**/*TestCase.java" - includes all of its subdirectories and all Java filenames that end with "TestCase".

Now these are just the problems I can see. It would be better to start with the cucumber-java-skeleton and transform it step by step into your project. If you run the tests after each change this should allow you to find the all problems that keep your project from working.