spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunita Arvind <>
Subject Integrate Spark Editor with Hue for source compiled installation of spark/spark-jobServer
Date Tue, 24 Jun 2014 16:04:57 GMT
Hello Experts,

I am attempting to integrate Spark Editor with Hue on CDH5.0.1. I have the
spark installation build manually from the sources for spark1.0.0. I am
able to integrate this with cloudera manager.

We have a 3 node VM cluster with CDH5.0.1
We requried spark1.0.0 due to some features in it, so I did a

 "yum remove spark-core spark-master spark-worker spark-python"

 of the default spark0.9.0 and compiled spark1.0.0 from source:

Downloaded the spark-trunk from

git clone
cd spark
SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true ./sbt/sbt assembly

The spark-assembly-1.0.0-SNAPSHOT-hadoop2.2.0.jar was built and spark by
itself seems to work well. I was even able to run a text file count.

Current attempt:
Referring to this article -
Now I am trying to add the Spark editor to Hue. AFAIK, this requires
git clone
cd spark-jobserver

This was successful after lot of struggle with the proxy settings. However,
is this the job Server itself? Will that mean the job Server has to be
manually started. I intend to have the spark editor show up in hue web UI
and I am no way close. Can some one please help?

Note, the 3 VMs are Linux CentOS. Not sure if setting something like can be
expected to work.:


Also, I have made the changes to vim .
/job-server/src/main/resources/application.conf as recommended, however, I
do not expect this to impact hue in any way.

Also, I intend to let the editor stay available, not spawn it everytime it
is required.

Thanks in advance.


View raw message