I'am going through the spark structured streaming documentation and kind of did not understood why there are restrictions regarding the session window. Please explain the below statements why is such:
Note that there are some restrictions when you use session window in streaming query, like below:
“Update mode” as output mode is not supported. There should be at least one column in addition to session_window in grouping key. For batch query, global window (only having session_window in grouping key) is supported.
By default, Spark does not perform partial aggregation for session window aggregation, since it requires additional sort in local partitions before grouping. It works better for the case there are only few number of input rows in same group key for each local partition, but for the case there are numerous input rows having same group key in local partition, doing partial aggregation can still increase the performance significantly despite additional sort.