spark-submit cammands throws error :Caused by: java.lang.IllegalStateException: java.net.MalformedURLException: unknown protocol: nested

39 views Asked by At

i am trying to run java springboot wth apache spark app, when i am generating the jar using maven clean install, i use that jar for spark-submit cammand, when when i try this cammand.

spark-submit --master local[*]  dr-0.0.1.jar -Dspring.profiles.active=local --add-exports --java.base/sun.nio.ch=ALL-UNNAMED --add-exports java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED 

i am getting this error:

C:\CODE\data\spark\appln_backend\target>spark-submit --master local[*]  dr-0.0.1.jar --add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-exports java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED -Dspring.profiles.active=local
24/02/27 23:40:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.springframework.boot.loader.net.protocol.jar.Handler.openConnection(Handler.java:46)
        at java.base/java.net.URL.openConnection(URL.java:1094)
        at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.getJarFile(JarUrlClassLoader.java:180)
        at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.definePackage(JarUrlClassLoader.java:143)
        at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.definePackageIfNecessary(JarUrlClassLoader.java:126)
        at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.loadClass(JarUrlClassLoader.java:99)
        at org.springframework.boot.loader.launch.LaunchedClassLoader.loadClass(LaunchedClassLoader.java:91)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:467)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:88)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53)
        at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalStateException: java.net.MalformedURLException: unknown protocol: nested
        at org.springframework.boot.loader.net.protocol.jar.JarUrlConnection.<clinit>(JarUrlConnection.java:67)
        ... 25 more
Caused by: java.net.MalformedURLException: unknown protocol: nested
        at java.base/java.net.URL.<init>(URL.java:681)
        at java.base/java.net.URL.<init>(URL.java:569)
        at java.base/java.net.URL.<init>(URL.java:516)
        at java.base/java.net.JarURLConnection.parseSpecs(JarURLConnection.java:175)
        at java.base/java.net.JarURLConnection.<init>(JarURLConnection.java:158)
        at org.springframework.boot.loader.net.protocol.jar.JarUrlConnection.<init>(JarUrlConnection.java:92)
        at org.springframework.boot.loader.net.protocol.jar.JarUrlConnection.<clinit>(JarUrlConnection.java:64)
        ... 25 more
24/02/27 23:40:51 INFO ShutdownHookManager: Shutdown hook called
24/02/27 23:40:51 INFO ShutdownHookManager: Deleting directory C:\Users\user\AppData\Local\Temp\spark-31e45f8c-e618-4323-8cad-cacd47edbddb

C:\CODE\data\spark\appln_backend\target>

if i include --class options in spark-submit cammand then this is error:

spark-submit --class com.group.dr.myApp --master local[*]  dr-0.0.1.jar

Error:
Error: Failed to load class com.group.dr.myApp.
24/02/27 21:02:35 INFO ShutdownHookManager: Shutdown hook called
24/02/27 21:02:35 INFO ShutdownHookManager: Deleting directory C:\Users\user\AppData\Local\Temp\spark-34d75945-ca3f-4a6a-8ffe-9d1253c8fe7f

i am using Versions : Java :17, Springboot : 3.2.0, Spark : 3.5.0, Scala : 2.13

my pom.xml looks like this

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.2.0</version>
        <relativePath />
        <!-- lookup parent from repository -->
    </parent>
    <groupId>com.group</groupId>
    <artifactId>dr</artifactId>
    <version>0.0.1</version>
    <packaging>jar</packaging>
    <name>data_app</name>
    <description>Springboot Project for Data app</description>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>17</java.version>
        <spring-cloud.version>2022.0.4</spring-cloud.version>
        <spark.version>3.5.0</spark.version>
        <scala.version>2.13</scala.version>
        <maven.compiler.source>${java.version}</maven.compiler.source>
        <maven.compiler.target>${java.version}</maven.compiler.target>
        <jakarta-servlet.version>4.0.3</jakarta-servlet.version>
        <jersey.version>2.36</jersey.version>
        <antlr4.version>4.9.3</antlr4.version>


    </properties>
    <dependencies>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>*</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jdbc</artifactId>
        </dependency>

        <dependency>
            <groupId>com.zaxxer</groupId>
            <artifactId>HikariCP</artifactId>
        </dependency>
        
        <dependency>
            <groupId>org.xerial</groupId>
            <artifactId>sqlite-jdbc</artifactId>
        </dependency>
        
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-csv</artifactId>
            <version>1.8</version>
        </dependency>
        <dependency>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
            <scope>runtime</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.springframework.data/spring-data-cassandra -->
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-cassandra</artifactId>

        </dependency>
    
        <!-- <dependency> <groupId>com.oracle.database.jdbc</groupId> <artifactId>ojdbc8</artifactId> 
            </dependency> -->
        <dependency>
            <groupId>com.oracle.database.jdbc</groupId>
            <artifactId>ojdbc10</artifactId>
            <version>19.22.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <dependency>
            <groupId>com.datastax.spark</groupId>
            <artifactId>spark-cassandra-connector_${scala.version}</artifactId>
            <version>3.4.1</version>
        </dependency>
        
        <!-- https://mvnrepository.com/artifact/com.esotericsoftware/kryo -->
        <dependency>
            <groupId>com.esotericsoftware</groupId>
            <artifactId>kryo</artifactId>
            <version>5.6.0</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <build>
        <plugins>


            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <!-- <version>3.8.8</version> -->
                <configuration>
                    <mainClass>com.group.dr.DataappApplication</mainClass>
                    <!-- <mainClass>${start.class}</mainClass> -->
                    <layout>JAR</layout>
                    <!-- 
                    <archive>
                        <manifest>

                            <addClassPath>true</addClassPath>
                            <mainClass>com.group.dr.DataappApplication</mainClass>
                        </manifest>
                    </archive>
                    <executable>true</executable> -->
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        
    
    </plugins>
</build>

</project>

and SparkCnfig.java is like this:

 @Bean
    public SparkSession sparkSession() {
        return SparkSession.builder()
                .appName(appName)
               .master(masterUri)
                .config("spark.cassandra.connection.host", "abc.com")
                .config("spark.cassandra.connection.port", "9042")
                .config("spark.cassandra.auth.username", "username")
                .config("spark.cassandra.auth.password", "pwd")
                .getOrCreate();
    }
0

There are 0 answers