Why should I really use Property-based testing if I already practice Example-based testing?

2.9k views Asked by At

One caveat that some developers argue about TDD with Example-based tests is the possible lack of every valid input handling.

Let's take a simple example where those developers could argue:

Make a program that add two numbers.

Bob, a developer starts writing the first test:
Given 1 + 3, then result is 4.
Implementation: def add(x: Int, y: Int) = 4
Nice, it passes.

Now second test:
Given 2 + 2, then result is 4. Implementation still the same: def add(x: Int, y: Int) = 4

So other developers would come and tell him:

"Now you see Bob that Example-based testing is not enough with your poor 2 examples !
You didn't test every output and looking at your actual implementation, it will fail for other inputs whose result is distinct from 4!
Now, listen to us and start writing some property-based tests to cover all valid inputs.
So let's start with commutativity test, associativity etc.. that are specific to the addition: properties of the addition !"

But but but.... Bob's TDD practice is really bad!
Indeed, he wanted to apply triangulation but he wrote a test that already succeeded, since the implementation hadn't to be altered!

To lead to Triangulation, one should write a test that fails. And triangulation is one of the master key of the practice of TDD.
It allows the main step: refactoring since you should manage two paths that lead to 2 distinct results!

=> As long as the tests get specific, the code gets more generic thanks to refactoring.

So my question is:
If we practice strict TDD with good practice of triangulation, what is the real benefit of using Property-based testing, asserting invariants that are in 99% of cases already cover by a good TDD?
Indeed, supposing that developers have a good IQ and build implementation that makes sense ;)

My example is taken from those slides.

2

There are 2 answers

0
piotrek On BEST ANSWER

property based testing is very helpful when edge cases are hard to find or there is so many of them that programmer can easily miss one. i used it for example when i was implementing hirschberg's algorithm. there is no obvious way to divide algorithm into smaller, trivial, easily TDD-testable pieces. and it's hard to hand-craft input that cover all possible algorithm paths. the best way is to generate large number of different input to cover all the paths. and when automatic checking find the bug, add that specific input to your regression tests

0
zhon On

If you practice TDD, you know it is a way of thinking about and doing design rather than than being about testing.

TDD originated as incremental, mostly, state based unit testing. Then Interaction-style (ie., mockist-style, or London-style) TDD changed how we think about code design code.

Property-based TDD has the opportunity to change the way we design code as well. Instead of generating examples to push our design forward, we use properties. This results in fewer tests that are more maintainable with better test coverage.

On the other hand, it is harder and requires more thinking than example based TDD.