How to generate smoke test cases for Java application using command line

Question

I wish to offer any new java based project to have basic/generic smoke testing as part of the CICD pipeline on github-actions or ansible.

Rather than having the developers write smoke test cases for their applications I wish to serve them with default test cases that are auto-generated by a tool or plugin.

One such plugin that I remember is diffblue

It has github CICD integration example step as well.

I have a few challenges using its CLI (free license) and what it offers is why I wish to explore other options.

My repo structure is :

myrepo
  |--src
      |--classes
            |--common
                 |--com
                     |--mybank
                          |--common
                               |--hello1.class
                               |--hello2.class
                               |--util
                                    |--hello3.class

I tried to generate testcase java files which should as per documentation be generated at

myrepo
  |--src
      |--test
            |--java
                                                     

1. dcover -classpath "D:\maven\bin\mvn.bat" com.mybank.common.hello1 does not generate any test cases java file, does not throw any error as per logs either in com.mybank.common or in src.test.java folders.

Not sure what is wrong in my attempt to use the cli for a single java file or entire java repo project.

2. I wish to explore alternate tools that may offer more services or are more popular.

Note: this project was build using ant build.xml

Kindly suggest.

Answer 1

First of all, your repo code is not a Maven project. For the standalone maven batch file to work, your project needs to be completely maven project containing pom.xml.

Note: DiffBlue won't work with Ant based projects.

So, consider migrating your Ant based codebase to Maven based codebase to use DiffBlue tool.

---

From Official DiffBlue Documentation:

Prerequisites:

1. Java 8 Update 351+, Java 11.0.17+ or Java 17.0.5+ are supported at this moment.

2. Only Maven 3.2.5+ or Gradle 4.9+ build tools are supported.

3. Any project (for use with Diffblue Cover) must compile and run with no failing unit tests. JUnit and TestNG testing frameworks are supported.

Example on how DiffBlue works with Maven:

Note: I'm using Mac to show how it works. The process is similar for Windows as well.

1. Read the documentation from here to setup DiffBlue CLI.

2. Execute dcover version to verify the DiffBlue installed or not.

   ➜  demo git:(master) ✗ dcover version
      2023.09.01-9200f07-licensed 
   

I have a spring-boot based maven project with this structure below.

.
├── mvnw
├── mvnw.cmd
├── pom.xml
├── src
│   ├── main
│   │   ├── java
│   │   │   └── com
│   │   │       └── example
│   │   │           └── demo
│   │   │               ├── SampleController.java
│   │   │              
│   │   │               
│   │   │               
│   │   │  
│   │   │               
│   │   └── resources
│   │       ├── application.yml
│   │       ├── static
│   │       └── templates
│   └── test
│       └── java
│           └── com
│               └── example
│                   └── demo
                        └── empty <no test classes>

3. I have a SampleController class with an endpoint divide which throws ArithmeticException:

   @RestController
   public class SampleController {
   
      @GetMapping("divide")
      public int divideByZero() throws ArithmeticException {
         int number = 2;
         return number / 0;
     }
   }
   

4. Enter root project directory with cd command.

Note: Execute the command from root directory itself as DiffBlue is smart enough to read all the classes and packages.

5. The command to generate the test is as follow :

   dcover create full-qualified-package-name.ClassName
   

Example: dcover create com.example.demo.SampleController

Current project structure contains the test class after dcover got executed:

.
├── mvnw
├── mvnw.cmd
├── pom.xml
├── src
│   ├── main
│   │   ├── java
│   │   │   └── com
│   │   │       └── example
│   │   │           └── demo
│   │   │               ├── SampleController.java
│   │   │              
│   │   │               
│   │   │               
│   │   │  
│   │   │               
│   │   └── resources
│   │       ├── application.yml
│   │       ├── static
│   │       └── templates
│   └── test
│       └── java
│           └── com
│               └── example
│                   └── demo
                        └── SampleControllerDiffblueTest.java

6. The test class generated from the above code is given below:

     package com.example.demo;
   
     import static org.junit.jupiter.api.Assertions.assertThrows;
     import static org.mockito.Mockito.mock;
     import org.junit.jupiter.api.Test;
   
     class SampleControllerDiffblueTest {
        /**
         * Method under test: {@link SampleController#divideByZero()}
         */
         @Test
         void testDivideByZero() {
          assertThrows(ArithmeticException.class, () -> (new SampleController()).divideByZero());
        }
     }
   

Console Logs:

INFO  Diffblue Cover 2023.09.01-9200f07-licensed
INFO
INFO  Detecting environment:
INFO  ----------------------
INFO  Detected Java version 17.0.8
INFO  Selected build system: Maven
INFO  Selected classpath from build system Maven
INFO  Selected test framework: JUnit Jupiter
INFO  Selected Java language compliance level: 17
INFO  Selected test validation: enabled
INFO  Prefixes for filtering classes/methods:
INFO  - com.example.demo.SampleController
INFO  Gathering methods to test...
........
INFO
INFO  Creating tests:
INFO  ---------------
INFO  [1/1] com.example.demo.SampleController
INFO  [1/1]   Tests created: 1
INFO
INFO  Summary:
INFO  --------
INFO  Total:                                               2 methods
INFO  1 test created for:                                  1 method
INFO
INFO  Tested indirectly via other methods:                 1 method
INFO      1 T005: Trivial constructor
INFO
INFO  Running validation command.
INFO  This can take a couple of minutes.........
INFO  1 created test was successfully validated.
INFO  Log file: /Users/anish/Downloads/demo/.diffblue/log/user-20230922T200532.log

INFO  Finished in 2m 12s

---

Also, You can try for other tools like Cucumber Framework which is an excellent framework for writing test classes via .feature file in human understand syntax and capable of generating test classes based on the .feature files (it uses Gherkin language).

Update: I found one tool called TackleTest-Unit which is capable of generating Java based Unit test cases. The main thing of this tool is that it supports Ant.

Github Repo: https://github.com/konveyor/tackle-test-generator-cli

From README.md :

TackleTest-Unit automatically generates unit test cases for Java applications. Each generated test case focuses on exercising the visible methods of an application class. Test generation can be done on a specific set of application classes or the entire application.

Among its many features, TackleTest-Unit:

• Supports automatic generation of test assertions: tests can be generated with or without assertions

• Combines different coverage metrics and generates coverage reports: the supported coverage metrics include conventional code coverage criteria, such as statement and branch coverage, as well as a novel type-based combinatorial coverage coverage of Java methods

• Accommodates developer-written unit test suites: it can perform coverage-driven augmentation and directed test generation to enhance the coverage achieved by a given test suite

• Supports multiple build tools: Maven, Gradle, and Ant

Read the docs for more. I feel that the tool looks promising. So, you can try it.