Skip to content

Embed a SparkContext within a Jetty web server

Notifications You must be signed in to change notification settings

spoddutur/spark-jetty-server

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

29 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Spark Jetty Server

This is a simple example demonstrating how to embed a SparkContext within a Jetty web server. This proved to be non-trivial as an understanding of how the Spark classpath is built is quite necessary to make this work. So far, this has only been tested using the Jetty maven plugin, but it should translate fairly easily to an actual jetty instance.

Architecture

image

Improvised

This project is originally forked from calrissian/spark-jetty-server. My contributions majorly is concentrated on below two chunks with an end goal of turning it into a plug-and-play from a laborious error-prone huge setup process:

  1. Simplified project setup - It needed classpath setup like SPARK_HOME, HADOOP_HOME etc and tweak externalClasspath property in maven pom file to point to the dependent jars. I've modified it and removed all such dependencies.
  2. Upgraded to latest ApacheSpark 2.1.x

Hence, running this project is now much more simplified - Just do mvn build and run!!

Building and running

The default build works with Spark 2.1.0 and Hadoop 2.7.0 but the versions are supplied through maven properties. To use a different Spark version, add -Dspark.version=<newVersion> to the maven command. To use a different Hadoop version, add -Dhadoop.version=<newVersion> to the maven command.

  1. Build the project from the root using mvn clean install.

  2. cd into the war/ directory and run mvn jetty:run.

  3. Once the webserver is up and running, navigate to http://localhost:8080/test and watch the result of a quick job.

Other thoughts

Perhaps the Spark documentation could better help users understand what's really going on behind the scenes with the classpaths of the various components involved (executor, master, driver, etc...). In this example, a SparkContext is instantiated by Spring's ApplicationContext at the deployment of the web application. This negotiates some resources ahead of time so that jobs can be run quickly without the overhead of having to negotiate those resources and fire up JVMs each time a job needs to be run. If this is not desired, it would be easy enough to have several different SparkContexts that run within the various different web application scopes (request, session, etc...).

Hopefully this will open up new doors for implementing different real-time query and data manipulation use-cases for Spark.

Releases

No releases published

Packages

No packages published

Languages

  • Scala 57.9%
  • Java 42.1%