Apache Spark pushes a bunch of information in event logs. How can I push my custom events into this event log?
Does Spark expose any APIs to do this? Or can I extend any existing class to do this?
Apache Spark pushes a bunch of information in event logs. How can I push my custom events into this event log?
Does Spark expose any APIs to do this? Or can I extend any existing class to do this?
You can extend "StreamingQueryListener" if you have a stream query and then add it to
spark.streams.addListener(new EventCollector(APP_NAME))
where
EventCollector(APP_NAME) extends StreamingQueryListener
and write your own methods
onQueryStarted
onQueryProgress
onQueryTerminated
methods and put custom code in it
Your class can extend
org.apache.spark.Logging
and use the help methods defined there:Note the warning on that trait: