How to provide a JUnit4 "conformance" test suite for an interface

Question

I would like to provide a test suite for an interface in our library so that clients that implement the interface can easily test their implementations for conformance with the specification of the interface.

A very simple example of this could look like this:

// Library code.
abstract class LibTest(impl: LibInterface) {
  @Test
  def myTest: Unit = {
    assertTrue(impl.getTrue())
  }
}

// Client code
class MyTest extends LibTest(new MyInterfaceImpl)

However, we need a couple of things more that turn out to be difficult with this approach. Notably:

• Parametrize some of our test cases internally (should not be visible to the client).
• Allow the client to pass properties of their implementation. Notably:
  • The set of supported features (based on which we want to enable/disable tests).
  • Supporting properties that tell the test how to interact with the implementation.

In our current prototype, we solve this by having suites per feature and constructors for parameters. Looks like this:

// Library code.
abstract class LibFeatureATest(impl: LibInterface, prop: String, paramA: Boolean)
abstract class LibFeatureBTest(impl: LibInterface, prop: String)
abstract class LibFeatureABTest(impl: LibInterface, prop: String, paramA: Boolean)

// Client code:
class FeatureATest(new MyInterfaceImpl, "my_prop", true)
class FeatureATest(new MyInterfaceImpl, "my_prop", false)
class FeatureBTest(new MyInterfaceImpl, "my_prop")

class FeatureABTest(new MyInterfaceImpl, "my_prop", true)
class FeatureABTest(new MyInterfaceImpl, "my_prop", false)

There are two main problems with this approach:

• The parameters for the tests need to be in the client.
• The client needs to instantiate tests for feature interactions.

Ideally we would like to have something like this in the client:

class MyTest extends TestSuite(new MyInterfaceImpl,
  prop = "my_prop",
  supportsA = true,
  supportsB = true)

The exact invocation syntax / type are secondary (abstract methods, parameters, annotations). Note however these additional requirements:

• Needs to be JUnit, we cannot depend on Scala specific testing frameworks.
• Needs to work in sbt, so some experimental runners (e.g. Enclosing) might not work.
• We would like to avoid unnecessary implementation inheritance.

How can we do this in JUnit4 in Scala on sbt?

Answer 1

You could implement JUnit Rules that provide parameterization and configuration, respectively.

The configuration (ie the properties given by the client) could be read from a file so the client can provide it separately without changing the test itself; the Rule would then enable or disable test cases depending on the config.

It's not clear what you mean by 'parameterized', but I'm sure you can use a Rule for that. They're essentially "Around" - aspects for test cases.

EDIT:

So this is how I imagine your interface:

interface ToTest {
    int getFeature1Value(); // let's say this needs to return 1 for compliance, if feature1 is supported
    int getFeature2Value(); // so this is a feature2 method, and needs to return 2
}

A complete Unit Test for this would be along the lines of

public class ComplianceTest {
    // the rule being applied, see below
    @Rule DisablingRule rule = new DisablingRule("feature_config.prop");

    @Autowired // so this would need to run in a DI environment
    ToTest sut;
    @FeatureSupport(condition = Features.Feature1) // see below
    @Test public void testFeature1() {
        assertEquals(1, sut.getFeature1Value());
    }
    @FeatureSupport(condition = Features.Feature2)
    @Test public void testFeature2() {
        assertEquals(2, sut.getFeature2Value());
    }
}

But IIUC you want the client to be able to specify which features are supported, and if they aren't, disable that test case. That's why I annotated the test cases with the FeatureSupport annotation, which would be a

@interface FeatureSupport {
    Features getCondition();
}

with a

enum Features {
    Feature1, Feature2
}

So that's the setup to write the Rule, which would be something like

class DisablingRule implements TestRule {
    Collection<Features> readFromFile = new ArrayList<>();
    public DisablingRule(String configfile) {
        // read config file to find out what features are supposed to be supported 
        // and add to member
        Files.readAllLines(Paths.get(configfile), Charset.UTF8).stream()
              .map(Feature1::valueOf)
              .forEach(readFromFile::add);
    }
    public Statement apply(Statement base, Description d) {
        return (isFeatureEnabled(d.getAnnotation(FeatureSupport.class)))
            ? base
            : () -> { 
                log(String.format("Test case %s disabled: feature not supported: %s", 
                    d.getDisplayName(), feature)); };
        }
    }
    private boolean isFeatureEnabled(FeatureSupport f) {
        return f != null || readFromFile.contains(f.getCondition());
    }
}

Answer 2

I missed your "ideally we would like to have something like this in the client", which makes things easier actually.

public interface MyTest<T> extends Test {
    public void setSut(T toTest);
    public void setProperties(Properties props);
}

public enum Features {
    Feature1(Feature1UnitTest::new), 
    Feature2(Feature2UnitTest::new);

    private Supplier<? extends Test> createTest;
    private Features(Supplier<? extends Test> sup) {
        createTest = sup;
    }
}

public class MyTestSuite<T> extends TestSuite {
    public MyTestSuite(T toTest, Properties testProperties, Features... supported) {
        for (Features f : supported) {
            MyTest<T> test = (MyTest<T>) f.createTest();
            test.setSut(toTest);
            test.setProperties(testProperties);
            addTest(test);
        }
    }
}