spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Mandel <mark.man...@gmail.com>
Subject Re: Confusion over how to deploy/run JAR files to a Spark Cluster
Date Thu, 02 Oct 2014 20:49:57 GMT
On 2 October 2014 21:38, Marius Soutier <mps.dev@gmail.com> wrote:

>
> On 02.10.2014, at 13:32, Mark Mandel <mark.mandel@gmail.com> wrote:
>
> How do I store a JAR on a cluster? Is that through storm-submit with a
> deploy mode of "cluster” ?
>
>
> Well, just upload it? scp, ftp, and so on. Ideally your build server would
> put it there.
>

Oooooh! *facepalm* When we say "upload" we actually are just saying "put it
on the master server". Right.



>  How do I run an already uploaded JAR with spark-submit? I can't seem to
> find any documentation for this?
>
>
> Just run spark-submit without any arguments, it will show you everything
> it supports.
>

Got it.


>
>
> I actually want to build a REST service for querying data at runtime from
> Spark (that is exactly my end use case), so having documentation on how to
> use this feature would be fantastic.
>
>
> There’s no real documentation on this because that’s not how it works
> unfortunately. You can search for a discussion I had on this group a couple
> of weeks ago. Typically you would store aggregates in a database. I ended
> up outputting Parquet files and reading them via SparkSQL. It works, but
> it’s not as fast as a database.
>
>
Yeah, sounds like I'm coming back to using Java->Scala interop with Spark
Job Server , which doing some reading doesn't sound quite as hard as I
originally thought it was.

Thanks for being so patient!

Mark


-- 
E: mark.mandel@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

2 Devs from Down Under Podcast
http://www.2ddu.com/

Mime
View raw message