Connecting Spark code from web application

1.2k views Asked by At

Can some one please let me know how to connect to Spark application from a web application. The Spark code whatever we are having is on Scala and we want to access the code from web application. If we can provide a small code snippet that will be helpful.

1

There are 1 answers

0
Sathish On

As mentioned in the comment section, try the following approaches,

spark-jobserver

spark-jobserver provides a RESTful interface for submitting and managing Apache Spark jobs, jars, and job contexts.

Approach mentioned by Jamal in spark-user-list

1) In the tomcat's classpath, add the spark distribution jar for spark code to be available at runtime

2) In the Web application project, add the spark distribution jar in the classpath ( Could be Java / Web project).

3) Setup the FAIR scheduling mode, which would help send parallel requests from web application to the spark cluster.

4) In the application startup, initialize the connection to the spark cluster. This is composed of creating the JavaSparkContext and making it available throughout the web application, in case this needs to be the only Driver Program required by the web application.

5) Using the JavaSpark Context, Create RDD's and make them available globally to the web application code.

6) invoke transformation / actions as required.