TechQA.

Cassandra with spark : java.io.IOException: Failed to open native connection to Cassandra at {127.0.0.1:9042} ::

1.5k views Asked by At

I have an application using Boot Strap running with cassandra 4.0, Cassandra java drive 4.11.1, spark 3.1.1 into ubuntu 20.4 with jdk 8_292 and python 3.6.

When I run a function that it call CQL by spark, the tomcat gave me the error bellow.

Stack trace:

  java.io.IOException: Failed to open native connection to Cassandra at {127.0.0.1:9042} :: Error instantiating class com.datastax.spark.connector.cql.LocalNodeFirstLoad>
            at com.datastax.spark.connector.cql.CassandraConnector$.createSession(CassandraConnector.scala:182) ~[spark-cassandra-connector_2.12-3.0.0.jar:3.0.0]
            at com.datastax.spark.connector.cql.CassandraConnector$.$anonfun$sessionCache$1(CassandraConnector.scala:170) ~[spark-cassandra-connector_2.12-3.0.0.jar:3.0.0]
            at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32) ~[spark-cassandra-connector_2.12-3.0.0.jar:3.0.0]
            at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69) ~[spark-cassandra-connector_2.12-3.0.0.jar:3.0.0]
            at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:57) ~[spark-cassandra-connector_2.12-3.0.0.jar:3.0.0]
            at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:90) ~[spark-cassandra-connector_2.12-3.0.0.jar:3.0.0]
            at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:112) ~[spark-cassandra-connector_2.12-3.0.0.jar:3.0.0]
            at com.datastax.spark.connector.datasource.CassandraCatalog$.com$datastax$spark$connector$datasource$CassandraCatalog$$getMetadata(CassandraCatalog.scala:455) >
            at com.datastax.spark.connector.datasource.CassandraCatalog$.getTableMetaData(CassandraCatalog.scala:421) ~[spark-cassandra-connector_2.12-3.0.0.jar:3.0.0]
            at org.apache.spark.sql.cassandra.DefaultSource.getTable(DefaultSource.scala:68) ~[spark-cassandra-connector_2.12-3.0.0.jar:3.0.0]
            at org.apache.spark.sql.cassandra.DefaultSource.inferSchema(DefaultSource.scala:72) ~[spark-cassandra-connector_2.12-3.0.0.jar:3.0.0]
            at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:81) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
            at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:296) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
            at scala.Option.map(Option.scala:230) ~[scala-library-2.12.11.jar:na]
            at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:266) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
            at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:226) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
            at br.com.genesis.configuration.DataSourceCassandraConfig.setParametros(DataSourceCassandraConfig.java:40) ~[classes/:2.4.2]
            at br.com.genesis.service.EstTaxaFalhaService.verificaAmostragemMinima(EstTaxaFalhaService.java:109) ~[classes/:2.4.2]
            at br.com.genesis.controller.estatistico.EstTaxaFalhaController.geraAmostraMinima(EstTaxaFalhaController.java:306) ~[classes/:2.4.2]

The cause by is:

Caused by: java.lang.IllegalArgumentException: Error instantiating class com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy (specified by basic.load-ba>
        at com.datastax.oss.driver.internal.core.util.Reflection.buildFromConfig(Reflection.java:253) ~[java-driver-core-4.11.1.jar:na]
        at com.datastax.oss.driver.internal.core.util.Reflection.buildFromConfigProfiles(Reflection.java:162) ~[java-driver-core-4.11.1.jar:na]
        at com.datastax.oss.driver.internal.core.context.DefaultDriverContext.buildLoadBalancingPolicies(DefaultDriverContext.java:338) ~[java-driver-core-4.11.1.jar:n>
        at com.datastax.oss.driver.internal.core.util.concurrent.LazyReference.get(LazyReference.java:55) ~[java-driver-core-4.11.1.jar:na]
        at com.datastax.oss.driver.internal.core.context.DefaultDriverContext.getLoadBalancingPolicies(DefaultDriverContext.java:687) ~[java-driver-core-4.11.1.jar:na]
        at com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded.init(DefaultSession.java:338) ~[java-driver-core-4.11.1.jar:na]
        at com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded.access$1100(DefaultSession.java:300) ~[java-driver-core-4.11.1.jar:na]
        at com.datastax.oss.driver.internal.core.session.DefaultSession.lambda$init$0(DefaultSession.java:146) ~[java-driver-core-4.11.1.jar:na]
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54) ~[netty-all-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
        ... 1 common frames omitted
Caused by: java.lang.NoSuchMethodError: com.datastax.oss.driver.internal.core.context.InternalDriverContext.getNodeFilter(Ljava/lang/String;)Ljava/util/function/Predic>
        at com.datastax.spark.connector.cql.LocalNodeFirstLoadBalancingPolicy.<init>(LocalNodeFirstLoadBalancingPolicy.scala:40) ~[spark-cassandra-connector-driver_2.1>
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_292]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_292]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_292]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_292]
        at com.datastax.oss.driver.internal.core.util.Reflection.buildFromConfig(Reflection.java:247) ~[java-driver-core-4.11.1.jar:na]
        ... 14 common frames omitted

First, I checked the nodetool status and it is ok:

~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load        Tokens  Owns (effective)  Host ID                               Rack
UN  127.0.0.1  556.74 KiB  16      100.0%            f457b508-1b91-456c-85bc-1a621c5c1d78  rack1

Then I checked the CQLSH prompt and it's ok :

~$cqlsh
Connected to SSP at 127.0.0.1:9042
[cqlsh 6.0.0 | Cassandra 4.0 | CQL spec 3.4.5 | Native protocol v5]
Use HELP for help.
cqlsh>

I'm using Cassandra as localhost (127.0.0.1). The .yaml and -env.sh are standard configurated.

I ran the command PS - ef and returned the cassandra's PID:

~$ ps -ef | grep cassandra
ubuntu     11139       1  1 18:28 ?        00:02:22 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -ea -da:net.openhft... -XX:+UseThreadPriorities -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseNUMA -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true -XX:ThreadPriorityPolicy=42 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xloggc:/var/log/cassandra/gc.log -Xms4002M -Xmx4002M -Xmn400M -XX:+UseCondCardMark -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler -javaagent:/usr/share/cassandra/lib/jamm-0.3.2.jar -Djava.rmi.server.hostname=127.0.0.1 -Dcassandra.jmx.local.port=7198 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password -Djava.library.path=/usr/share/cassandra/lib/sigar-bin -XX:OnOutOfMemoryError=kill -9 %p -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=/var/lib/cassandra -cp /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.8.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-7.1.jar:/usr/share/cassandra/lib/caffeine-2.3.5.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.11.0-shaded.jar:/usr/share/cassandra/lib/chronicle-bytes-2.20.111.jar:/usr/share/cassandra/lib/chronicle-core-2.20.126.jar:/usr/share/cassandra/lib/chronicle-queue-5.20.123.jar:/usr/share/cassandra/lib/chronicle-threads-2.20.111.jar:/usr/share/cassandra/lib/chronicle-wire-2.20.117.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.9.jar:/usr/share/cassandra/lib/commons-lang3-3.11.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/ecj-4.6.1.jar:/usr/share/cassandra/lib/guava-27.0-jre.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.8.1.jar:/usr/share/cassandra/lib/j2objc-annotations-1.3.jar:/usr/share/cassandra/lib/jackson-annotations-2.9.10.jar:/usr/share/cassandra/lib/jackson-core-2.9.10.jar:/usr/share/cassandra/lib/jackson-databind-2.9.10.8.jar:/usr/share/cassandra/lib/jamm-0.3.2.jar:/usr/share/cassandra/lib/java-cup-runtime-11b-20160615.jar:/usr/share/cassandra/lib/javax.inject-1.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.25.jar:/usr/share/cassandra/lib/jcommander-1.30.jar:/usr/share/cassandra/lib/jctools-core-3.1.0.jar:/usr/share/cassandra/lib/jflex-1.8.2.jar:/usr/share/cassandra/lib/jna-5.6.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jvm-attach-api-1.5.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.25.jar:/usr/share/cassandra/lib/logback-classic-1.2.3.jar:/usr/share/cassandra/lib/logback-core-1.2.3.jar:/usr/share/cassandra/lib/lz4-java-1.7.1.jar:/usr/share/cassandra/lib/metrics-core-3.1.5.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.5.jar:/usr/share/cassandra/lib/metrics-logback-3.1.5.jar:/usr/share/cassandra/lib/mxdump-0.14.jar:/usr/share/cassandra/lib/netty-all-4.1.58.Final.jar:/usr/share/cassandra/lib/netty-tcnative-boringssl-static-2.0.36.Final.jar:/usr/share/cassandra/lib/ohc-core-0.5.1.jar:/usr/share/cassandra/lib/ohc-core-j8-0.5.1.jar:/usr/share/cassandra/lib/psjava-0.1.19.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.3.jar:/usr/share/cassandra/lib/reporter-config3-3.0.3.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/sjk-cli-0.14.jar:/usr/share/cassandra/lib/sjk-core-0.14.jar:/usr/share/cassandra/lib/sjk-json-0.14.jar:/usr/share/cassandra/lib/sjk-stacktrace-0.14.jar:/usr/share/cassandra/lib/slf4j-api-1.7.25.jar:/usr/share/cassandra/lib/snakeyaml-1.26.jar:/usr/share/cassandra/lib/snappy-java-1.1.2.6.jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/zstd-jni-1.3.8-5.jar:/usr/share/cassandra/lib/jsr223/*/*.jar:/usr/share/cassandra/apache-cassandra-4.0.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/fqltool.jar:/usr/share/cassandra/stress.jar: org.apache.cassandra.service.CassandraDaemon

The tomcat 9 connected with the cassandra as well.

POM.xml

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.4.2</version>
    <relativePath /> <!-- lookup parent from repository -->
</parent>
<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-web -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-tomcat -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-tomcat</artifactId>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot</artifactId>
    </dependency>
    <dependency>
        <groupId>javax.persistence</groupId>
        <artifactId>javax.persistence-api</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-data-cassandra -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-cassandra</artifactId>
    </dependency>
    <dependency>
        <groupId>javax.servlet</groupId>
        <artifactId>javax.servlet-api</artifactId>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-thymeleaf</artifactId>
    </dependency>
    <dependency>
        <groupId>org.python</groupId>
        <artifactId>jython</artifactId>
        <version>2.7.2</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector -->
    <dependency>
        <groupId>com.datastax.spark</groupId>
        <artifactId>spark-cassandra-connector_2.12</artifactId>
        <version>3.0.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>3.1.1</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>3.1.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-exec</artifactId>
        <version>1.3</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.springframework/spring-core -->
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-core</artifactId>
    </dependency>
    <dependency>
        <groupId>org.javassist</groupId>
        <artifactId>javassist</artifactId>
        <version>3.27.0-GA</version>
    </dependency>
    <dependency>
        <groupId>javax.xml.bind</groupId>
        <artifactId>jaxb-api</artifactId>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-math3 -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-math3</artifactId>
        <version>3.6.1</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.yaml/snakeyaml -->
    <dependency>
        <groupId>org.yaml</groupId>
        <artifactId>snakeyaml</artifactId>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.google.code.gson/gson -->
    <dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-rng-parent</artifactId>
        <version>1.3</version>
        <type>pom</type>
    </dependency>
    <dependency>
        <groupId>org.hamcrest</groupId>
        <artifactId>hamcrest-core</artifactId>
        <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.json/json -->
    <dependency>
        <groupId>org.json</groupId>
        <artifactId>json</artifactId>
        <version>20200518</version>
    </dependency>
    <dependency>
        <groupId>com.opencsv</groupId>
        <artifactId>opencsv</artifactId>
        <version>4.2</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.datastax.oss/java-driver-core -->
    <dependency>
        <groupId>com.datastax.oss</groupId>
        <artifactId>java-driver-core</artifactId>
        <version>4.11.1</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.datastax.oss/java-driver-query-builder -->
    <dependency>
        <groupId>com.datastax.oss</groupId>
        <artifactId>java-driver-query-builder</artifactId>
    </dependency>
    <!-- https://mvnrepository.com/artifact/joda-time/joda-time -->
    <dependency>
        <groupId>joda-time</groupId>
        <artifactId>joda-time</artifactId>
        <version>2.10.10</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-configuration-processor</artifactId>
    </dependency>
    <dependency>
        <groupId>org.codehaus.janino</groupId>
        <artifactId>commons-compiler</artifactId>
    </dependency>
    <dependency>
        <groupId>org.codehaus.janino</groupId>
        <artifactId>janino</artifactId>
    </dependency>
    <!-- https://mvnrepository.com/artifact/com.datastax.oss/native-protocol -->
    <dependency>
        <groupId>com.datastax.oss</groupId>
        <artifactId>native-protocol</artifactId>
        <version>1.5.0</version>
    </dependency>
    <dependency>
        <groupId>calculosSSP</groupId>
        <artifactId>CalculosSSP</artifactId>
        <version>1.2.8</version>
    </dependency>
</dependencies>

Does anyone have an idea about what is happening?

best,

java apache-spark ubuntu cassandra cassandra-4.0
Original Q&A
1

There are 1 answers

1
Luciana Oliveira On BEST ANSWER

I openned two JIRA to understand this problem. See the links below:

  1. https://datastax-oss.atlassian.net/browse/SPARKC-652
  2. https://datastax-oss.atlassian.net/browse/JAVA-2945

Related Questions in JAVA

Related Questions in APACHE-SPARK

Related Questions in UBUNTU

Related Questions in CASSANDRA

Related Questions in CASSANDRA-4.0

Popular Questions

Popular Tags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Trending Questions