How to implement EXISTS condition as like SQL in spark Dataframe

Question

I am curious to know, how can i implement sql like exists clause in spark Dataframe way.

Answer 1

LEFT SEMI JOIN is equivalent to the EXISTS function in Spark.

val cityDF= Seq(("Delhi","India"),("Kolkata","India"),("Mumbai","India"),("Nairobi","Kenya"),("Colombo","Srilanka")).toDF("City","Country")

val CodeDF= Seq(("011","Delhi"),("022","Mumbai"),("033","Kolkata"),("044","Chennai")).toDF("Code","City")

val finalDF= cityDF.join(CodeDF, cityDF("City") === CodeDF("City"), "left_semi")

Answer 2

If the data to be compared is small like a broadcasted list then you can use -

df.filter(col("columnName").isin(list...))