SparkLauncher constructor not working

Question

I need to run a Spark job from thread launched from a Web Application running on Liberty (17.0.0.2). This is deployed on a Docker container.

I am able to get this working by launching the spark (2.2.0) job through spark-submit script launched from the java code. However, I want to remove the script invocation and use SparkLauncher. When I do this, the code hangs on SparkLauncher constructor. The constructor does not return anything blocking the whole work. I don't see logs after the invocation of the constructor in code below.

private static Map<String, String> env = new HashMap<String, String>();
static {
    env.put("SPARK_HOME", "/Spark");
    env.put("JAVA_HOME", "/opt/ibm/java-x86_64-80/jre");
    env.put("SPARK_PRINT_LAUNCH_COMMAND","1");
}
private static final String SPARK_MASTER_URL = "local[*]";

public SparkAppHandle launch(String appName, String appJar, String sparkClass, String[] args)
        throws IOException, InterruptedException {
    logger.logp(Level.INFO, CLASS, "launch", " About to create SparkLauncher ");
    SparkLauncher launchPad = new SparkLauncher(env);

    logger.logp(Level.INFO, CLASS, "launch", " About to set Spark App Name ");
    launchPad = launchPad.setAppName(appName);

Any clue on how to debug this further? I could not find logs with any information to debug this issue.

Answer 1

Do you call the launch method of SparkLauncher ? It would return Process, and you need to capture the output of this process to see what's going on

Answer 2

It was a class path issue; some jars were missing. However, somehow there was no information in the log to debug this, very strange. After ensuring all relevant jars in the classpath it started working.