i am trying to run java springboot wth apache spark app, when i am generating the jar using maven clean install, i use that jar for spark-submit cammand, when when i try this cammand.
spark-submit --master local[*] dr-0.0.1.jar -Dspring.profiles.active=local --add-exports --java.base/sun.nio.ch=ALL-UNNAMED --add-exports java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED
i am getting this error:
C:\CODE\data\spark\appln_backend\target>spark-submit --master local[*] dr-0.0.1.jar --add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-exports java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED -Dspring.profiles.active=local
24/02/27 23:40:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.springframework.boot.loader.net.protocol.jar.Handler.openConnection(Handler.java:46)
at java.base/java.net.URL.openConnection(URL.java:1094)
at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.getJarFile(JarUrlClassLoader.java:180)
at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.definePackage(JarUrlClassLoader.java:143)
at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.definePackageIfNecessary(JarUrlClassLoader.java:126)
at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.loadClass(JarUrlClassLoader.java:99)
at org.springframework.boot.loader.launch.LaunchedClassLoader.loadClass(LaunchedClassLoader.java:91)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:467)
at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:88)
at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53)
at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalStateException: java.net.MalformedURLException: unknown protocol: nested
at org.springframework.boot.loader.net.protocol.jar.JarUrlConnection.<clinit>(JarUrlConnection.java:67)
... 25 more
Caused by: java.net.MalformedURLException: unknown protocol: nested
at java.base/java.net.URL.<init>(URL.java:681)
at java.base/java.net.URL.<init>(URL.java:569)
at java.base/java.net.URL.<init>(URL.java:516)
at java.base/java.net.JarURLConnection.parseSpecs(JarURLConnection.java:175)
at java.base/java.net.JarURLConnection.<init>(JarURLConnection.java:158)
at org.springframework.boot.loader.net.protocol.jar.JarUrlConnection.<init>(JarUrlConnection.java:92)
at org.springframework.boot.loader.net.protocol.jar.JarUrlConnection.<clinit>(JarUrlConnection.java:64)
... 25 more
24/02/27 23:40:51 INFO ShutdownHookManager: Shutdown hook called
24/02/27 23:40:51 INFO ShutdownHookManager: Deleting directory C:\Users\user\AppData\Local\Temp\spark-31e45f8c-e618-4323-8cad-cacd47edbddb
C:\CODE\data\spark\appln_backend\target>
if i include --class options in spark-submit cammand then this is error:
spark-submit --class com.group.dr.myApp --master local[*] dr-0.0.1.jar
Error:
Error: Failed to load class com.group.dr.myApp.
24/02/27 21:02:35 INFO ShutdownHookManager: Shutdown hook called
24/02/27 21:02:35 INFO ShutdownHookManager: Deleting directory C:\Users\user\AppData\Local\Temp\spark-34d75945-ca3f-4a6a-8ffe-9d1253c8fe7f
i am using Versions : Java :17, Springboot : 3.2.0, Spark : 3.5.0, Scala : 2.13
my pom.xml looks like this
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.2.0</version>
<relativePath />
<!-- lookup parent from repository -->
</parent>
<groupId>com.group</groupId>
<artifactId>dr</artifactId>
<version>0.0.1</version>
<packaging>jar</packaging>
<name>data_app</name>
<description>Springboot Project for Data app</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>17</java.version>
<spring-cloud.version>2022.0.4</spring-cloud.version>
<spark.version>3.5.0</spark.version>
<scala.version>2.13</scala.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
<jakarta-servlet.version>4.0.3</jakarta-servlet.version>
<jersey.version>2.36</jersey.version>
<antlr4.version>4.9.3</antlr4.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>log4j</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
</dependency>
<dependency>
<groupId>org.xerial</groupId>
<artifactId>sqlite-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
<version>1.8</version>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.data/spring-data-cassandra -->
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-cassandra</artifactId>
</dependency>
<!-- <dependency> <groupId>com.oracle.database.jdbc</groupId> <artifactId>ojdbc8</artifactId>
</dependency> -->
<dependency>
<groupId>com.oracle.database.jdbc</groupId>
<artifactId>ojdbc10</artifactId>
<version>19.22.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_${scala.version}</artifactId>
<version>3.4.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.esotericsoftware/kryo -->
<dependency>
<groupId>com.esotericsoftware</groupId>
<artifactId>kryo</artifactId>
<version>5.6.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<!-- <version>3.8.8</version> -->
<configuration>
<mainClass>com.group.dr.DataappApplication</mainClass>
<!-- <mainClass>${start.class}</mainClass> -->
<layout>JAR</layout>
<!--
<archive>
<manifest>
<addClassPath>true</addClassPath>
<mainClass>com.group.dr.DataappApplication</mainClass>
</manifest>
</archive>
<executable>true</executable> -->
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
and SparkCnfig.java is like this:
@Bean
public SparkSession sparkSession() {
return SparkSession.builder()
.appName(appName)
.master(masterUri)
.config("spark.cassandra.connection.host", "abc.com")
.config("spark.cassandra.connection.port", "9042")
.config("spark.cassandra.auth.username", "username")
.config("spark.cassandra.auth.password", "pwd")
.getOrCreate();
}