How to unit test Spark Streaming code?

Question

I use the latest Spark 1.6.0.

Looked at another stackoverflow post How can I make Spark Streaming count the words in a file in a unit test?

I am trying to use the sample @ https://gist.github.com/emres/67b4eae86fa92df69f61 have for writing a sample unit test for spark. I intend to write a unit test after that for my actual spark application. However, I am unable to use the example @ https://gist.github.com/emres/67b4eae86fa92df69f61. This gives me error

java.lang.IllegalStateException: Adding new inputs, transformations, and output operations after starting a context is not supported at oracle.security.ti.reportgenerator.test.StarterAppTest.testCountWords(StarterAppTest.java:62)

Any ideas how I can resolve this problem?

Answer 1

You cannot do wordCounts.print() on JavaPairDStream wordCounts after you have started streaming context.