I created a dummy custom QueryExecutionListener(given below) according to the information here https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-ExecutionListenerManager.html and here https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/exercises/spark-exercise-custom-scheduler-listener.html.
The custom listener just had some print statements. The listener was added via the configuration property spark.sql.queryExecutionListeners . However i do not see any of my logging statements in the console of spark submit command. Also there are no errors as a result of spark submit. I can see the properties that are set in by using "spark.sqlContext.getAllConfs"
It looks like the onSuccess and onFailure methods are not at all getting called.
Has anyone ever successfully created a custom query execution listener and "registered" it using the conf properties?
//code for the customlistener is given below:
class LineageListener extends QueryExecutionListener with Logging {
override def onSuccess(funcName: String, qe: QueryExecution, durationNs: Long): Unit = {
log.info("The function ${funcName} succeeded")
val sparkContext = qe.sparkSession.sparkContext
log info("App name: ${sparkContext.appName} and id is ${sparkContext.applicationId}") }
override def onFailure(funcName: String, qe: QueryExecution, durationNs: Long): Unit = {
log.info("The function ${funcName} succeeded")
val sparkContext = qe.sparkSession.sparkContext
log info("App name: ${sparkContext.appName} and id is ${sparkContext.applicationId}") }
Note: My spark version is 2.2.1
It depends on how you called the spark submit.If everything is at local you must see since all drivers share the same console. If you submit by yarn (where workers are other machines) you can see logs via spark ui or other log viewing tools.