Why are test marked as passed in spite of failures

Question

I'm using Scalatest/Scalacheck with custom generators. I observe that tests are marked as success even if some tests failed. In below example test "should add processing timestamp" was Falsified. Yet sbt test passed.

+ OK, passed 100 tests.
[info] - should add product info to event 
[info] - should not alter rest of event
+ OK, passed 100 tests.
! Falsified after 0 passed tests.
> ARG_0: List("([B@27d10fe1,...)")
> ARG_0_ORIGINAL: List("([B@3c8057ce,...)")
[info] - should add processing timestamp
[info] ScalaTest
[info] Run completed in 4 seconds, 792 milliseconds.
[info] Total number of tests run: 3
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 3, Failed 0, Errors 0, Passed 3
[success] Total time: 8 s, completed Sep 11, 2017 6:54:28 PM

Why is test not failing??

UPDATE: sbt 0.13, scalatest 3.0.1, scalacheck 1.13.4 and the test case is

  it should "add processing timestamp" in {
    Prop.forAll(sizedGen(TestInputGen.Generate)) { in => 
      val out = processor.input(in)

      out.forall(o => {
        val outTS = o._2.get("timestamps")
        (outTS.getModule() == "PrimaryProcessor")
      })
    }
  }.check

Answer 1

Since you are using ScalaTest style of property based testing as opposed to the ScalaCheck style of property testing, your properties need to return a Matcher or an Assertion instead of a Boolean expression.

From the documentation:

In the ScalaTest property style you use the word whenever instead of ==> and either an assertion or matcher expression instead of a boolean expression

Below in an example you can use to test this. The test tests that the length of 2 concatenated strings is always bigger than the length of any of the strings used in the concatenation. This should fail when both strings are empty

Compiles but tests pass because we are using a Boolean expression

import org.scalatest.prop.PropertyChecks
import org.scalatest.{FlatSpec, Matchers}

final class StringSpec extends FlatSpec with Matchers with PropertyChecks {

  behavior of "String"

  it should "concatenate" in {
    forAll { (a: String, b: String) =>
      (a + b).length > a.length && (a + b).length > b.length
    }
  }
}

Compiles and fails as expected because it uses `Matchers

import org.scalatest.prop.PropertyChecks
import org.scalatest.{Assertion, FlatSpec, Matchers}

final class StringSpec extends FlatSpec with Matchers with PropertyChecks {

  behavior of "String"

  it should "concatenate" in {
    forAll { (a: String, b: String) =>
      (a + b).length should be > a.length
      (a + b).length should be >= b.length
    }
  }
}