On 2 October 2014 21:38, Marius Soutier <mps.dev@gmail.com> wrote:

On 02.10.2014, at 13:32, Mark Mandel <mark.mandel@gmail.com> wrote:

How do I store a JAR on a cluster? Is that through storm-submit with a deploy mode of "cluster” ?

Well, just upload it? scp, ftp, and so on. Ideally your build server would put it there.

Oooooh! *facepalm* When we say "upload" we actually are just saying "put it on the master server". Right.

 How do I run an already uploaded JAR with spark-submit? I can't seem to find any documentation for this? 

Just run spark-submit without any arguments, it will show you everything it supports.

Got it.

I actually want to build a REST service for querying data at runtime from Spark (that is exactly my end use case), so having documentation on how to use this feature would be fantastic.

There’s no real documentation on this because that’s not how it works unfortunately. You can search for a discussion I had on this group a couple of weeks ago. Typically you would store aggregates in a database. I ended up outputting Parquet files and reading them via SparkSQL. It works, but it’s not as fast as a database.

Yeah, sounds like I'm coming back to using Java->Scala interop with Spark Job Server , which doing some reading doesn't sound quite as hard as I originally thought it was.

Thanks for being so patient!


E: mark.mandel@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

2 Devs from Down Under Podcast