Flakiness in tests on Android using LiveData, RxJava/RxKotlin and Spek

Question

Setup:

In our project (at work - I cannot post real code), we have implemented clean MVVM. Views communicate with ViewModels via LiveData. ViewModel hosts two kinds of use cases: 'action use cases' to do something, and 'state updater use cases'. Backward communication is asynchronous (in terms of action reaction). It's not like an API call where you get the result from the call. It's BLE, so after writing the characteristic there will be a notification characteristic we listen to. So we use a lot of Rx to update the state. It's in Kotlin.

ViewModel:

@PerFragment
class SomeViewModel @Inject constructor(private val someActionUseCase: SomeActionUseCase,
                                        someUpdateStateUseCase: SomeUpdateStateUseCase) : ViewModel() {

    private val someState = MutableLiveData<SomeState>()

    private val stateSubscription: Disposable

    // region Lifecycle
    init {
        stateSubscription = someUpdateStateUseCase.state()
                .subscribeIoObserveMain() // extension function
                .subscribe { newState ->
                    someState.value = newState
                })
    }

    override fun onCleared() {
        stateSubscription.dispose()

        super.onCleared()
    }
    // endregion

    // region Public Functions
    fun someState() = someState

    fun someAction(someValue: Boolean) {
        val someNewValue = if (someValue) "This" else "That"

        someActionUseCase.someAction(someNewValue)
    }
    // endregion
}

Update state use case:

@Singleton
class UpdateSomeStateUseCase @Inject constructor(
            private var state: SomeState = initialState) {

    private val statePublisher: PublishProcessor<SomeState> = 
            PublishProcessor.create()

    fun update(state: SomeState) {
        this.state = state

        statePublisher.onNext(state)
    }

    fun state(): Observable<SomeState> = statePublisher.toObservable()
                                                       .startWith(state)
}

We are using Spek for unit tests.

@RunWith(JUnitPlatform::class)
class SomeViewModelTest : SubjectSpek<SomeViewModel>({

    setRxSchedulersTrampolineOnMain()

    var mockSomeActionUseCase = mock<SomeActionUseCase>()
    var mockSomeUpdateStateUseCase = mock<SomeUpdateStateUseCase>()

    var liveState = MutableLiveData<SomeState>()

    val initialState = SomeState(initialValue)
    val newState = SomeState(newValue)

    val behaviorSubject = BehaviorSubject.createDefault(initialState)

    subject {
        mockSomeActionUseCase = mock()
        mockSomeUpdateStateUseCase = mock()

        whenever(mockSomeUpdateStateUseCase.state()).thenReturn(behaviorSubject)

        SomeViewModel(mockSomeActionUseCase, mockSomeUpdateStateUseCase).apply {
            liveState = state() as MutableLiveData<SomeState>
        }
    }

    beforeGroup { setTestRxAndLiveData() }
    afterGroup { resetTestRxAndLiveData() }

    context("some screen") {
        given("the action to open the screen") {
            on("screen opened") {
                subject
                behaviorSubject.startWith(initialState)

                it("displays the initial state") {
                    assertEquals(liveState.value, initialState)
                }
            }
        }

        given("some setup") {
            on("some action") {
                it("does something") {
                    subject.doSomething(someValue)

                    verify(mockSomeUpdateStateUseCase).someAction(someOtherValue)
                }
            }

            on("action updating the state") {
                it("displays new state") {
                    behaviorSubject.onNext(newState)

                    assertEquals(liveState.value, newState)
                }
            }
        }
    }
}

At first we were using an Observable instead of the BehaviorSubject:

var observable = Observable.just(initialState)
...
whenever(mockSomeUpdateStateUseCase.state()).thenReturn(observable)
...
observable = Observable.just(newState)
assertEquals(liveState.value, newState)

instead of the:

val behaviorSubject = BehaviorSubject.createDefault(initialState)
...
whenever(mockSomeUpdateStateUseCase.state()).thenReturn(behaviorSubject)
...
behaviorSubject.onNext(newState)
assertEquals(liveState.value, newState)

but the unit test were being flaky. Mostly they would pass (always when ran in isolation), but sometime they would fail when running the whole suit. Thinking it is to do with asynchronous nature of the Rx we moved to BehaviourSubject to be able to control when the onNext() happens. Test are now passing when we run them from AndroidStudio on the local machine, but they are still flaky on the build machine. Restarting the build often makes them pass.

The tests which fail are always the ones where we assert the value of LiveData. So the suspects are LiveData, Rx, Spek or their combination.

Question: Did anyone have similar experiences writing unit tests with LiveData, using Spek or maybe Rx, and did you find ways to write them which solve these flakiness issues?

....................

Helper and extension functions used:

fun instantTaskExecutorRuleStart() =
        ArchTaskExecutor.getInstance().setDelegate(object : TaskExecutor() {
            override fun executeOnDiskIO(runnable: Runnable) {
                runnable.run()
            }

            override fun isMainThread(): Boolean {
                return true
            }

            override fun postToMainThread(runnable: Runnable) {
                runnable.run()
            }
        })

fun instantTaskExecutorRuleFinish() = ArchTaskExecutor.getInstance().setDelegate(null)

fun setRxSchedulersTrampolineOnMain() = RxAndroidPlugins.setInitMainThreadSchedulerHandler { Schedulers.trampoline() }

fun setTestRxAndLiveData() {
    setRxSchedulersTrampolineOnMain()
    instantTaskExecutorRuleStart()
}

fun resetTestRxAndLiveData() {
    RxAndroidPlugins.reset()
    instantTaskExecutorRuleFinish()
}

fun <T> Observable<T>.subscribeIoObserveMain(): Observable<T> =
        subscribeOnIoThread().observeOnMainThread()

fun <T> Observable<T>.subscribeOnIoThread(): Observable<T> = subscribeOn(Schedulers.io())

fun <T> Observable<T>.observeOnMainThread(): Observable<T> =
        observeOn(AndroidSchedulers.mainThread())

Answer 1

The issue is not with LiveData; it is the more common problem - singletons. Here the Update...StateUseCases had to be singletons; otherwise if observers got a different instance they would have a different PublishProcessor and would not get what was published.

There is a test for each Update...StateUseCases and there is a test for each ViewModel into which Update...StateUseCases is injected (well indirectly via the ...StateObserver).

The state exists within the Update...StateUseCases, and since it is a singleton, it gets changed in both tests and they use the same instance becoming dependent on each other.

Firstly try to avoid using singletons if possible.

If not, reset the state after each test group.

Answer 2

I didn't used Speck for unit-testing. I've used java unit-test platform and it works perfect with Rx & LiveData, but you have to keep in mind one thing. Rx & LiveData are async and you can't do something like someObserver.subscribe{}, someObserver.doSmth{}, assert{} this will work sometimes but it's not the correct way to do it.

For Rx there's TestObservers for observing Rx events. Something like:

@Test
public void testMethod() {
   TestObserver<SomeObject> observer = new TestObserver()
   someClass.doSomethingThatReturnsObserver().subscribe(observer)
   observer.assertError(...)
   // or
   observer.awaitTerminalEvent(1, TimeUnit.SECONDS)
   observer.assertValue(somethingReturnedForOnNext)
}

For LiveData also, you'll have to use CountDownLatch to wait for LiveData execution. Something like this:

@Test
public void someLiveDataTest() {
   CountDownLatch latch = new CountDownLatch(1); // if you want to check one time exec
   somethingTahtReturnsLiveData.observeForever(params -> {
      /// you can take the params value here
      latch.countDown();
   }
   //trigger live data here
   ....
   latch.await(1, TimeUnit.SECONDS)
   assert(...)
} 

Using this approach your test should run ok in any order on any machine. Also the wait time for latch & terminal event should be as low as possible, the tests should run fast.

Note1: The code is in JAVA but you can change it easily in kotlin.

Note2: Singleton are the biggest enemy of unit-testing ;). (With static methods by their side).