spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Jain <>
Subject Re: Confusion over how to deploy/run JAR files to a Spark Cluster
Date Thu, 02 Oct 2014 14:00:46 GMT
Hello Mark,

I am no expert but I can answer some of your questions.

On Oct 2, 2014 2:15 AM, "Mark Mandel" <> wrote:
> Hi,
> So I'm super confused about how to take my Spark code and actually deploy
and run it on a cluster.
> Let's assume I'm writing in Java, and we'll take a simple example such
and this is a process I want to be running quite regularly (say more than
once a minute).
> From the documentation (, it reads
as if I need to create a jar from the above code, and every time I want to
run this code, I use ./bin/spark-submit to upload it to the cluster, which
would then run it straight away.
> This would mean that every time I want to run my process, I need to have
a .jar file travel over the network? Is this correct? (seems like this
would be very slow? I should try it however).
> Doing some digging around the JavaDocs, I can see that the
Java/SparkContext has the option to .addJar()'s , but I can't see any
documentation that actually outlines how this can be used?  If someone can
point me towards an article or tutorial on how this is meant to work, I'd
greatly appreciate it.
> It would seem like I could write a simple process that ran, quite
probably on the same machine as master, that added a Jar through the
SparkContext... but then, how to run the code from that Jar?
> Or is the Jar include the code that I would run, that would then create
the SparkContext that would addJar itself? (now my head hurts).
On a single node program this works. The way I think of this is, though I
might be wrong here, you specify spark configurations in your driver
program. This is where you specify master, the jar containing all
dependencies, memory & serialization parameters. When you do a SparkContext
in this driver program the embedded(?) Spark instance runs and picks up the
jar that you specified. The code that you wrote is added like any other

If you have all configuration provided through SparkConf along with the
setJars, you could do a sbt 'runMain className args[]' to invoke your
> Would Spark also be smart enough to know that the JAR was already
uploaded, if addJar was called once it had already been uploaded?
> I'm not seeing this shown in the examples either.
> I'm really excited by what I see in Spark, but I am totally confused by
how to actually get code up on Spark and make it run, and nothing I read
seems to explain this aspect very well (at least to my thick head).
> I have seen:, but from
initial review, it looks like it will only work with Scala, (because you
need to use the ScalaJob trait), and I have a Java dependency.
You can implement this as an interface in Java. You can pass the
SparkContext as a parameter to JavaSparkContext. The only non intuitive
effort here is to return a valid job you will need to write this - return

> Any help on this aspect would be greatly appreciated!
> Mark
> --
> E:
> T:
> W:
> 2 Devs from Down Under Podcast

View raw message