Why should I use triangulation instead of just randomized values in my tests?

Question

Since I started learning and using TDD I was wondering something about the "triangulation technique": why bother writing multiple tests when we could write one that use randomized values?

Today, I stumbled upon this article, which explains the "triangulation technique". Then I posted a question to the author about this "randomization over triangulation" questionment. I am very interested in his view on this, but I was also wanting to gather as many point of views as possible on this matter.

What he explains is very interesting, because I prefered randomized values as I thought that using one or two or even three differents values do not "prove" that my system is robust. But his insight on the refactoring phase that should also avoid duplication between the test and the implementation is the key.

That being said, I still find it a bit useless. I mean: using this technique, one has to write multiple similar tests (at least 2), an implementation that will evolve, and then do a refactoring to all of this to remove all tests but one. The remaining test will not express the complexity of the final solution, as there is no direct link between it and the implementation. It does not endorse its role of "documenting" the implementation. It cannot be used to lead to the implementation once again.

With a randomized value, on the other hand, I directly have to adjust my implementation to the fact that the value is not known. I cannot use a simple constant to make the test pass. I do not have to write multiple tests that will be removed. I do not have to be rigorous enough to not forget to do this test-implementation de-duplication refactoring. And my test do express the correct need served by the implementation (i.e.: given "any" two numbers, I return the sum).

So, over your experience, do you see any advantage of triangulation over the use of randomized values?

Answer 1

When Kent Beck first introduced the triangulation metaphor, he wrote:

I've tried Triangulation, but the steps don't seem valuable to me, so I quickly lose motivation.

---

So, over your experience, do you see any advantage of triangulation over the use of randomized values?

This is a somewhat complicated question to unpack.

There are a few demonstrations of TDD with property based tests on line.

• Seb Rose
• Nat Pryce

When I try it, however, I quickly run into a couple problems

1. Unless you're trying to solve a trivial problem, defining properties is hard.

2. It's not at all clear that tests are the right tool to use to ensure that properties hold. For instance, Jim Coplien wrote that most unit tests should be assertions.

Part of the tension here is that, while property driven tests may be better for testing, the intention of TDD is in the realm of analysis and design.

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence -- Beck, 2008

The checks that we create in TDD are by the developers for the developers. So unless testing with random data provides design insights that aren't otherwise available, it seems to be a priority of wearing the "wrong" hat.

Answer 2

It's not about triangulation Vs property based testing! It's about how to use both, the problem is that the article uses a very trivial example, in which property based testing makes 100% sense from the start.

Use triangulation while building the solution, in order to go one step at a time. If you work on a technology that has a good REPL (Haskell, Clojure, etc), this can be done directly from the REPL. This helps to build confidence that the solution works with the basic scenarios we throw at it.

When the solution is done, write one (or more) property based tests that would cover multiple of the other basic test.

Tests should also be seen as documentation of how the System Under Test (SUT) works. In some cases it might be good to have some non-property based test to describe how the SUT works. And a mini warning on the complexity of the code that creates random values, if that is too complex it has a chance of making the tests very difficult to understand.

Answer 3

After decades of TDD, I've long since reached the conclusion that no single technique fits all problems. As the tired cliché goes, use the right tool for the job.

Sometimes TDD with triangulation works well, but in other cases, it really doesn't work well. Here's an example where example-driven development didn't work well, but property-based testing did: When properties are easier than examples

Other people on this page have stated that property-based testing may work for simple problems, but perhaps not for 'real' problems. I beg to differ. The above link is an example of a situation where property-based testing got me out of a situation I couldn't easily triangulate my way out of.

Another, simpler problem where property-based testing is more suitable than triangulation is the Diamond kata. That, at least, is my experience, having tried to solve it both ways.

Why should I use triangulation instead of just randomized values in my tests?

You shouldn't. Sometimes triangulation is the best approach, and sometimes, other techniques (like property-based testing) works better.

Sometimes you can combine methods. You can, for example, still write properties before you implement the System Under Tests, so this is still TDD. You may also call it triangulation, but the scale is more granular, since a single property will typically run with hundreds of randomly-generated values.

Here's an example of combining TDD and PBT: Property-based testing is not the same as partition testing

The remaining test will not express the complexity of the final solution, as there is no direct link between it and the implementation. It does not endorse its role of "documenting" the implementation. It cannot be used to lead to the implementation once again.

Perhaps, but the implementation is not the goal. In fact, one may argue that the implementation is, well, an implementation detail. What matters is that automated tests exercise the behaviour of the System Under Test (SUT).

If you imagine that you delete all the SUT code and re-implement it from the test suite, does it matter that the new code looks different from the old code?

Perhaps the new code is better. That's the underlying idea behind refactoring: Changing the structure of the code without changing the behaviour of the system.