BeanCreationException: Error creating bean with name 'defaultValidator' on Spark-Submit

Question

Created a spark structured streaming application using spring boot. The bootRun works fine, but when deploying the same jar using "spark-submit", gives me the following error :

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultValidator' defined in class path resource [org/springframework/boot/autoconfigure/validation/ValidationAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.NoSuchMethodError: javax.validation.BootstrapConfiguration.getClockProviderClassName()Ljava/lang/String;

Following is my dependencies list from build.gradle

compile("org.springframework.boot:spring-boot-starter-security")
compile('org.apache.kafka:kafka-streams')
compileOnly('org.projectlombok:lombok:1.18.2')
testCompile('org.springframework.boot:spring-boot-starter-test')
testCompile('org.springframework.security:spring-security-test')
compile('org.apache.spark:spark-sql_2.11:2.3.1')
compile('org.apache.spark:spark-streaming-kafka-0-10_2.11:2.3.1')
compile('org.apache.spark:spark-streaming_2.11:2.3.1')
compile('org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.1')
compile group: 'com.google.protobuf', name: 'protobuf-java', version: '3.6.1'
compile('org.springframework.boot:spring-boot-starter-web')
compile group: 'javax.validation', name: 'validation-api', version: '2.0.1.Final'
testCompile group: 'junit', name: 'junit', version: '4.12'
compile "redis.clients:jedis:2.9.0"
compile group: 'org.eclipse', name: 'yasson', version: '1.0.1'
compile (group: 'org.glassfish', name: 'javax.json', version: '1.1.2')

Help Required! Any suggestions?

Answer 1

Reason: Spark has earlier version of 'javax.validation:validation-api', without "getClockProviderClassName" method.

Options like "userClassPathFirst" can help: Classpath resolution between spark uber jar and spark-submit --jars when similar classes exist in both