Linked Questions

Popular Questions

Spark: subtract values in same DataSet row

Asked by At

Given the following dataset:

| title | start | end
| bla   | 10    | 30

I would like to find the difference (start - end) between the two numbers and set them into a new column, so that it looks like:

| title | time_spent |
 | bla   | 20 |

The data is of type Dataset<Row>
dataset = dataset.withColumn("millis spent: ", col("end") - col("start")).as("Time spent");
I expected this to work, as i saw in this question, but it does ot, maybe because that thread is about DataFrames and not DataSets, or maybe because Scala allows it where as in Java is illegal?

Related Questions