spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marius Soutier <>
Subject Re: Confusion over how to deploy/run JAR files to a Spark Cluster
Date Thu, 02 Oct 2014 11:38:19 GMT

On 02.10.2014, at 13:32, Mark Mandel <> wrote:

> How do I store a JAR on a cluster? Is that through storm-submit with a deploy mode of
"cluster” ?

Well, just upload it? scp, ftp, and so on. Ideally your build server would put it there.

>  How do I run an already uploaded JAR with spark-submit? I can't seem to find any documentation
for this? 

Just run spark-submit without any arguments, it will show you everything it supports.

> I actually want to build a REST service for querying data at runtime from Spark (that
is exactly my end use case), so having documentation on how to use this feature would be fantastic.

There’s no real documentation on this because that’s not how it works unfortunately. You
can search for a discussion I had on this group a couple of weeks ago. Typically you would
store aggregates in a database. I ended up outputting Parquet files and reading them via SparkSQL.
It works, but it’s not as fast as a database.

View raw message