Trouble finding Spark master with DSE 4.7

622 views Asked by At

I upgraded to DSE 4.7, and now the Spark master address seems to have been reset. When I try to run spark using dse spark, I get:

java.io.IOException: Spark Master address cannot be retrieved. This really should not be happening with DSE 4.7+ unless your cluster is over 50% down or booted up in the last minute.
    at com.datastax.bdp.plugin.SparkPlugin.getMasterAddress(SparkPlugin.java:257)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
    at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)

Same error when I run dsetool sparkmaster. Is it possible to manually set the master in a conf somewhere? There seem to be many configuration file Any help appreciated.

0

There are 0 answers