Py4JJavaError: An error occurred while calling z:is.hail.backend.spark.SparkBackend.apply

137 views Asked by At

I am trying to go through the tutorial of Hail by using Jupyter Notebook installed in Anaconda. However, I am getting the error from the first step. Here are the error codes:

Py4JJavaError: An error occurred while calling z:is.hail.backend.spark.SparkBackend.apply.
: java.lang.ExceptionInInitializerError
    at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(ByteArrayMethods.java:56)
    at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes$lzycompute(MemoryManager.scala:264)
    at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes(MemoryManager.scala:254)
    at org.apache.spark.memory.MemoryManager.$anonfun$pageSizeBytes$1(MemoryManager.scala:273)
    at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.memory.MemoryManager.<init>(MemoryManager.scala:273)
    at org.apache.spark.memory.UnifiedMemoryManager.<init>(UnifiedMemoryManager.scala:58)
    at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:207)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:320)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:464)
    at is.hail.backend.spark.SparkBackend$.configureAndCreateSparkContext(SparkBackend.scala:145)
    at is.hail.backend.spark.SparkBackend$.apply(SparkBackend.scala:237)
    at is.hail.backend.spark.SparkBackend.apply(SparkBackend.scala)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalStateException: java.lang.NoSuchMethodException: java.nio.DirectByteBuffer.<init>(long,int)
    at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:113)
    ... 26 more
Caused by: java.lang.NoSuchMethodException: java.nio.DirectByteBuffer.<init>(long,int)
    at java.base/java.lang.Class.getConstructor0(Class.java:3761)
    at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2930)
    at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:71)
    ... 26 more

How do I resolve this error?

I have tried to reinstall Java, hail, and Spark through pip install, but nothing really resolved this issue.

0

There are 0 answers