How to calculate time difference with nscala_time in Spark

Question

I'm trying to calculate the time difference of T1 and T2. My code in spark-shell is as follow:

scala> import com.github.nscala_time.time.Imports._
import com.github.nscala_time.time.Imports._
scala> import org.joda.time.DateTime
import org.joda.time.DateTime

scala> val T1 = DateTime.parse("2014-10-12T10:32:32")
T1: org.joda.time.DateTime = 2014-10-12T10:32:32.000+08:00
scala> val T2 = DateTime.parse("2014-10-12T10:33:32")
T2: org.joda.time.DateTime = 2014-10-12T10:33:32.000+08:00
scala> val diff = (T1 to T2).millis/1000
diff: Long = 60

everything works fine. So I spark-submit the same code with sbt:

my dependency file is as follow:

submit code:

spark-shell --class "YourApp" --jars /usr/local/spark/lib_third/nscala-time-master/target/scala-2.10/nscala-time_2.10-2.15.0-SNAPSHOT.jar ~/Spark_nscala2/target/scala-2.10/YourApp_2.10-1.0.jar

However,it went wrong:

Exception in thread "main" java.lang.NoSuchMethodError: com.github.nscala_time.time.Imports$.RichReadableInstant(Lorg/joda/time/ReadableInstant;)Lcom/github/nscala_time/time/RichReadableInstant;
    at YourApp$.main(YourApp.scala:15)
    at YourApp.main(YourApp.scala)

exception shows the problem comes from this sentence:

val diff = (T1 to T2).millis/1000

it doesn't make sense to me,the same code works in spark-shell but fails when submitted. Can someone tell me where the problem is?Thanks in advance!

Answer 1

I figure it out myself.I tried another version of nscala_time jar,and changed the dependency accordingly,the problem is fixed.

Answer 2

try to add your third party jar to --driver-class-path as well as --jars