Recently I've updated the version of org.apache.spark.spark-core and org.apache.spark.spark-sql to version 2.4.3 in my Scala application.
Since then, my application is not able to send emails anymore. The library used for the email functionality is org.apache.commons.commons-email, version 1.4. There is no much in the logs about the issue, only the stacktrace below:
org.apache.commons.mail.Email.sendMimeMessage(Email.java:1421)
org.apache.commons.mail.Email.send(Email.java:1448)
(...)
I checked the smtp host and everything is up and running (and also sending emails). Other applications with previous version of spark (2.0.1) works perfectly.
UPDATE
Even after returning the library versions to 2.0.1, the issue persisted. The issue only get solved when I also return the ${SPARK_HOME}/bin/spark-submit back to version 2.0.1 or any version < 2.1.
Is there any known issue related to Spark-submit update, that conflicts with the library org.apache.commons.commons-email?
Thank you.