spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lbustelo <>
Subject IBM open-sources Spark Kernel
Date Mon, 15 Dec 2014 15:19:45 GMT
This was posted on the Dev list, but it is very relevant to the user list as

We are happy to announce a developer preview of the Spark Kernel which 
enables remote applications to dynamically interact with Spark. You can 
think of the Spark Kernel as a remote Spark Shell that uses the IPython 
notebook interface to provide a common entrypoint for any application. The 
Spark Kernel obviates the need to submit jars using spark-submit, and can 
replace the existing Spark Shell. 

You can try out the Spark Kernel today by installing it from our github 
repo at To help you get a demo 
environment up and running quickly, the repository also includes a 
Dockerfile and a Vagrantfile to build a Spark Kernel container and connect 
to it from an IPython notebook. 

We have included a number of documents with the project to help explain it 
and provide how-to information: 

* A high-level overview of the Spark Kernel and its client library (

* README ( - 
building and testing the kernel, and deployment options including building 
the Docker container and packaging the kernel. 

* IPython instructions ( - 
setting up the development version of IPython and connecting a Spark 

* Client library tutorial ( - 
building and using the client library to connect to a Spark Kernel. 

* Magics documentation ( - the 
magics in the kernel and how to write your own. 

We think the Spark Kernel will be useful for developing applications for 
Spark, and we are making it available with the intention of improving these 
capabilities within the context of the Spark community ( We will continue to 
develop the codebase and welcome your comments and suggestions. 


Chip Senkbeil 
IBM Emerging Technology Software Engineer

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message