I've a streaming application in structured streaming , I want to get all the records within a defined window (this is a event time window), when I am trying to use groupby(window) and use aggregate as count , I'm not able to fetch all the columns.
val windowSpec = window($"timestamp", " minute","1 minutes")
// Perform the grouping and aggregation
val windowedDF = timestampedDF
.withColumn("window", windowSpec)
.groupBy($"window")
.count()
How can I get all the records within a defined window and without using an agg. functions ?
I tried Using flatMapGroupsWithState with forEachBatch this solution but stuck in same I need to use forEachBatch for some computations on Micro batches.