Error when using SparkJob with NamedRddSupport

Question

Goal is to create the following on a local instance of Spark JobServer:

object foo extends SparkJob with NamedRddSupport

Question: How can I fix the following error which happens on every job:

{
  "status": "ERROR",
  "result": {
    "message": "Ask timed out on [Actor[akka://JobServer/user/context-supervisor/439b2467-spark.jobserver.genderPrediction#884262439]] after [10000 ms]",
    "errorClass": "akka.pattern.AskTimeoutException",
    "stack: ["akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)", "akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)", "scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)", "scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)", "akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)", "akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)", "akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)", "akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)", "java.lang.Thread.run(Thread.java:745)"]
  }
}

A more detailed error description by the Spark JobServer:

job-server[ERROR] Exception in thread "pool-100-thread-1" java.lang.AbstractMethodError: spark.jobserver.genderPrediction$.namedObjectsPrivate()Ljava/util/concurrent/atomic/AtomicReference;
job-server[ERROR]   at spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:248)
job-server[ERROR]   at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
job-server[ERROR]   at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
job-server[ERROR]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
job-server[ERROR]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
job-server[ERROR]   at java.lang.Thread.run(Thread.java:745)

In case somebody wants to see the code:

package spark.jobserver
import org.apache.spark.SparkContext._
import org.apache.spark.{SparkContext}
import com.typesafe.config.{Config, ConfigFactory}
import collection.JavaConversions._
import scala.io.Source

object genderPrediction extends SparkJob with NamedRddSupport
{
  // Main function
  def main(args: scala.Array[String]) 
  {
    val sc = new SparkContext()
    sc.hadoopConfiguration.set("fs.tachyon.impl", "tachyon.hadoop.TFS")
    val config = ConfigFactory.parseString("")
    val results = runJob(sc, config)
  }

  def validate(sc: SparkContext, config: Config): SparkJobValidation = {SparkJobValid}

  def runJob(sc: SparkContext, config: Config): Any =
  {
    return "ok";
  }
}

Version information: Spark is 1.5.0 - SparkJobServer is latest version

Thank you all very much in advance!

Answer 1

Adding more explanation to @noorul 's answer

It seems like you compiled the code with an old version of SJS and you are running it with the latest.

NamedObjects were recently added. You are getting AbstractMethodError because your server expects NamedObjects support and you didn't compile the code with that.

Also: you don't need the main method there since it won't be executed by SJS.

Answer 2

Ensure that your.compile and run time library versions of dependent packages are same.