We are at a road block trying to understand why Spark 1.3.1 doesn't work for a colleague of mine on his Windows 7 laptop. I have pretty much the same setup and everything works fine for me.
I searched for the error message but still didn't find a resolution.
Here is the exception message (after running spark 1.3.1 vanilla installation prebuilt for Hadoop 2.4)
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://sparkDriver/deadLetters), Path(/)
]
at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at scala.concurrent.impl.Promise$DefaultPromise.scala$concurrent$impl$Promise$DefaultPromise$$dispatchOrAddCallb
ack(Promise.scala:280)
at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:270)
at akka.actor.ActorSelection.resolveOne(ActorSelection.scala:63)
at akka.actor.ActorSelection.resolveOne(ActorSelection.scala:80)
at org.apache.spark.util.AkkaUtils$.makeDriverRef(AkkaUtils.scala:221)
at org.apache.spark.executor.Executor.startDriverHeartbeater(Executor.scala:393)
at org.apache.spark.executor.Executor.<init>(Executor.scala:119)
at org.apache.spark.scheduler.local.LocalActor.<init>(LocalBackend.scala:58)
at org.apache.spark.scheduler.local.LocalBackend$$anonfun$start$1.apply(LocalBackend.scala:107)
at org.apache.spark.scheduler.local.LocalBackend$$anonfun$start$1.apply(LocalBackend.scala:107)
at akka.actor.TypedCreatorFunctionConsumer.produce(Props.scala:343)
at akka.actor.Props.newActor(Props.scala:252)
at akka.actor.ActorCell.newActor(ActorCell.scala:552)
at akka.actor.ActorCell.create(ActorCell.scala:578)
... 9 more
Related Issues:
Searching for this issue online didn't yield a lot of results, which seems to point that this is a very rare and specific problem:
I have seen this error mentioned, but for Linux, not windows: http://apache-spark-user-list.1001560.n3.nabble.com/Actor-not-found-td22265.html
This one also doesn't seem to offer any resolution: https://groups.google.com/a/lists.datastax.com/forum/#!topic/spark-connector-user/UqCYeUpgGCU
My assumption is that this is related to some resolving / IP conflicts etc, but I'm not sure.
More details
- JDK 1.7 64 bit, Windows 7 64 bit, Spark 1.3.1 prebuilt for Hadoop 2.4
- We ruled out all firewall related issues, we looked at all blocked traffic and it was not there
- We tried "run as an administrator" with no luck.
- We tried both Spark submit and spark shell, the most straightforward Spark "Hellow World" didn't work.
- We did get the UI at localhost:4040, the job is marked as started, but is waiting forever (e.g. even
sc.parallelize(List(1,2,3)).count()never finishes - No other errors were found in logs
- One difference that I did notice between my system and my friend's: when I do ping localhost, I get 127.0.0.1, when he does it he gets ::1 not sure if it's related, but I saw an issue about spark having problems with ipv6, and saw it was resolved only in 1.4, is that related? https://issues.apache.org/jira/browse/SPARK-6440
I'm sure it's a network / security / permissions related issue, but we can't seem to be able to pinpoint it
Any ideas where to look next?
Upgrading to Spark 1.4.0 seems to have resolved that issue.
It could be related to https://issues.apache.org/jira/browse/SPARK-6440 but can't tell for sure.