spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Mandel <>
Subject Re: Confusion over how to deploy/run JAR files to a Spark Cluster
Date Thu, 02 Oct 2014 20:49:57 GMT
On 2 October 2014 21:38, Marius Soutier <> wrote:

> On 02.10.2014, at 13:32, Mark Mandel <> wrote:
> How do I store a JAR on a cluster? Is that through storm-submit with a
> deploy mode of "cluster” ?
> Well, just upload it? scp, ftp, and so on. Ideally your build server would
> put it there.

Oooooh! *facepalm* When we say "upload" we actually are just saying "put it
on the master server". Right.

>  How do I run an already uploaded JAR with spark-submit? I can't seem to
> find any documentation for this?
> Just run spark-submit without any arguments, it will show you everything
> it supports.

Got it.

> I actually want to build a REST service for querying data at runtime from
> Spark (that is exactly my end use case), so having documentation on how to
> use this feature would be fantastic.
> There’s no real documentation on this because that’s not how it works
> unfortunately. You can search for a discussion I had on this group a couple
> of weeks ago. Typically you would store aggregates in a database. I ended
> up outputting Parquet files and reading them via SparkSQL. It works, but
> it’s not as fast as a database.
Yeah, sounds like I'm coming back to using Java->Scala interop with Spark
Job Server , which doing some reading doesn't sound quite as hard as I
originally thought it was.

Thanks for being so patient!



2 Devs from Down Under Podcast

View raw message