How to implement EXISTS condition as like SQL in spark Dataframe

7.6k views Asked by At

I am curious to know, how can i implement sql like exists clause in spark Dataframe way.

2

There are 2 answers

2
venus On BEST ANSWER

LEFT SEMI JOIN is equivalent to the EXISTS function in Spark.

val cityDF= Seq(("Delhi","India"),("Kolkata","India"),("Mumbai","India"),("Nairobi","Kenya"),("Colombo","Srilanka")).toDF("City","Country")

df1

val CodeDF= Seq(("011","Delhi"),("022","Mumbai"),("033","Kolkata"),("044","Chennai")).toDF("Code","City")

df2

val finalDF= cityDF.join(CodeDF, cityDF("City") === CodeDF("City"), "left_semi")

df3

0
Salim On

If the data to be compared is small like a broadcasted list then you can use -

df.filter(col("columnName").isin(list...))