Combine two spark udf issue

Question

I'm using Spark 1.6 with scala; I have to compute the duration which is the difference between end time and start time. I've tried this:

val msc3 = rddsql.withColumn("Duration",($"EndTime")-($"StartTime"))

I want to add another condition: when the end time and the start time are equal, the duration should be set to 1 instead of 0. How to do it ?

Answer 1

You don't need UDFs at all, you can simply do it using when and otherwise

rddsql.withColumn("Duration",when($"EndTime" === $"StartTime", 1).otherwise($"EndTime" - $"StartTime"))

Answer 2

You can also do it with 'Case When' and SparkSQL

rddsql.createOrReplaceTempView("rddsql")
spark.sql("select CASE WHEN (EndTime-StartTime = 0) THEN 1 ELSE EndTime-StartTime END as Duration from rddsql") //spark is SparkSession