I'm receiving streaming data from Kafka, which I'm reading as a dataframe with Structured Spark Streaming.
The problem is that I need to perform multiple aggregations on the same column and non-time-based window operations with that results.
AFAIK that's still not possible in Spark Structured Streaming, so I want to start a Spark batch job triggered after some time.
How could I achive that? Is there any way to start a python script like with spark submit?