How to pass FsCheck Test Correctly

Question

let list p = if List.contains " " p || List.contains null p then false else true

I have such a function to check if the list is well formatted or not. The list shouldn't have an empty string and nulls. I don't get what I am missing since Check.Verbose list returns falsifiable output.

How should I approach the problem?

Answer 1

I think you don't quite understand FsCheck yet. When you do Check.Verbose someFunction, FsCheck generates a bunch of random input for your function, and fails if the function ever returns false. The idea is that the function you pass to Check.Verbose should be a property that will always be true no matter what the input is. For example, if you reverse a list twice then it should return the original list no matter what the original list was. This property is usually expressed as follows:

let revTwiceIsSameList (lst : int list) =
    List.rev (List.rev lst) = lst

Check.Verbose revTwiceIsSameList  // This will pass

Your function, on the other hand, is a good, useful function that checks whether a list is well-formed in your data model... but it's not a property in the sense that FsCheck uses the term (that is, a function that should always return true no matter what the input is). To make an FsCheck-style property, you want to write a function that looks generally like:

let verifyMyFunc (input : string list) =
    if (input is well-formed) then  // TODO: Figure out how to check that
        myFunc input = true
    else
        myFunc input = false

Check.Verbose verifyMyFunc

(Note that I've named your function myFunc instead of list, because as a general rule, you should never name a function list. The name list is a data type (e.g., string list or int list), and if you name a function list, you'll just confuse yourself later on when the same name has two different meanings.)

Now, the problem here is: how do you write the "input is well-formed" part of my verifyMyFunc example? You can't just use your function to check it, because that would be testing your function against itself, which is not a useful test. (The test would essentially become "myFunc input = myFunc input", which would always return true even if your function had a bug in it — unless your function returned random input, of course). So you'd have to write another function to check if the input is well-formed, and here the problem is that the function you've written is the best, most correct way to check for well-formed input. If you wrote another function to check, it would boil down to not (List.contains "" || List.contains null) in the end, and again, you'd be essentially checking your function against itself.

In this specific case, I don't think FsCheck is the right tool for the job, because your function is so simple. Is this a homework assignment, where your instructor is requiring you to use FsCheck? Or are you trying to learn FsCheck on your own, and using this exercise to teach yourself FsCheck? If it's the former, then I'd suggest pointing your instructor to this question and see what he says about my answer. If it's the latter, then I'd suggest finding some slightly more complicated function to use to learn FsCheck. A useful function here would be one where you can find some property that should always be true, like in the List.rev example (reversing a list twice should restore the original list, so that's a useful property to test with). Or if you're having trouble finding an always-true property, at least find a function that you can implement in at least two different ways, so that you can use FsCheck to check that both implementations return the same result for any given input.

Answer 2

Adding to @rmunn's excellent answer:

if you wanted to test myFunc (yes I also renamed your list function) you could do it by creating some fixed cases that you already know the answer to, like:

let myFunc p = if List.contains " " p || List.contains null p then false else true

let tests =
    testList "myFunc" [
        testCase "empty list"    <| fun()-> "empty" |> Expect.isTrue  (myFunc [      ])
        testCase "nonempty list" <| fun()-> "hi"    |> Expect.isTrue  (myFunc [ "hi" ])
        testCase "null case"     <| fun()-> "null"  |> Expect.isFalse (myFunc [ null ])
        testCase "empty string"  <| fun()-> "\"\""  |> Expect.isFalse (myFunc [ ""   ])
    ]

Tests.runTests config tests

Here I am using a testing library called Expecto.

If you run this you would see one of the tests fails:

Failed! myFunc/empty string: "". Actual value was true but had expected it to be false.

because your original function has a bug; it checks for space " " instead of empty string "".

After you fix it all tests pass:

4 tests run in 00:00:00.0105346 for myFunc – 4 passed, 0 ignored, 0 failed, 0 errored. Success!

At this point you checked only 4 simple and obvious cases with zero or one element each. Many times functions fail when fed more complex data. The problem is how many more test cases can you add? The possibilities are literally infinite!

FsCheck

This is where FsCheck can help you. With FsCheck you can check for properties (or rules) that should always be true. It takes a little bit of creativity to think of good ones to test for and granted, sometimes it is not easy.

In your case we can test for concatenation. The rule would be like this:

• If two lists are concatenated the result of MyFunc applied to the concatenation should be true if both lists are well formed and false if any of them is malformed.

You can express that as a function this way:

let myFuncConcatenation l1 l2 = myFunc (l1 @ l2) = (myFunc l1 && myFunc l2)

l1 @ l2 is the concatenation of both lists.

Now if you call FsCheck:

FsCheck.Verbose myFuncConcatenation

It tries a 100 different combinations trying to make it fail but in the end it gives you the Ok:

0:
["X"]
["^"; ""]
1:
["C"; ""; "M"]
[]
2:
[""; ""; ""]
[""; null; ""; ""]
3:
...
Ok, passed 100 tests.

This does not necessarily mean your function is correct, there still could be a failing combination that FsCheck did not try or it could be wrong in a different way. But it is a pretty good indication that it is correct in terms of the concatenation property.

Testing for the concatenation property with FsCheck actually allowed us to call myFunc 300 times with different values and prove that it did not crash or returned an unexpected value.

FsCheck does not replace case by case testing, it complements it:

Notice that if you had run FsCheck.Verbose myFuncConcatenation over the original function, which had a bug, it would still pass. The reason is the bug was independent of the concatenation property. This means that you should always have the case by case testing where you check the most important cases and you can complement that with FsCheck to test other situations.

Here are other properties you can check, these test the two false conditions independently:

let myFuncHasNulls l = if List.contains null l then myFunc l = false else true
let myFuncHasEmpty l = if List.contains ""   l then myFunc l = false else true

Check.Quick myFuncHasNulls
Check.Quick myFuncHasEmpty

// Ok, passed 100 tests.
// Ok, passed 100 tests.