How can I find out why giving the wrong spec doesn't cause a blowup in my tests?

Question

Parts of my system are specced out really well, but when I change one of the predicates to be something obviously wrong I noticed all my tests still pass and I don't get the usual blowup from spec I've come to rely on.

I can't figure out why this is happening, and I certainly can't reproduce it starting from lein new test.

Is there a way I can get spec.test to give me a warning when it can't find a spec, for debug purposes, rather than assuming I didn't want to spec out this part of my system? Can it perhaps help me in some other way with debugging this situation?

Answer 1

Here is a minimal reproduction:

(ns test.core
  (:require [clojure.spec :as s]))

(defn my-specced-fn [x]
  x)

(s/fdef my-specced-fn
        :args (s/cat :arg int?))

(ns test.core-test
  (:require [clojure
             [test :refer :all]]
            [test.core :as core]
            [clojure.spec.test :as spec-test]))

(spec-test/instrument)

(deftest my-specced-fn-test
  (is (= 1 (core/my-specced-fn 1))))

This test passes initially. I would then go edit test.core, change the schema and re-evaluate test.core. After changing the schema with a predicate like string? the tests should fail, but it keeps passing. To solve the problem re-evaluate the test namespace (specifically the call to instrument).

Answer 2

spec should error if you try to use a spec that's not defined.

There is no way currently to have it tell you about things that aren't spec'ed. To do so would require instrumenting (replacing) all vars and adding that check.

For your particular problem, if you have a spec that you're changing, I would search for who is using that predicate and then try testing each thing that uses those specs or the original predicate.

One thing that trips people up sometimes is that stest/instrument only checks the :args specs of functions, not the :ret or :fn specs (which are only used by stest/check).