spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Surendranauth Hiraman <suren.hira...@velos.io>
Subject Re: Play framework
Date Thu, 16 Oct 2014 16:42:45 GMT
Mohammed,

Jumping in for Daniel, we actually address the configuration issue by
pulling values from environment variables or command line options. Maybe
that may handle at least some of your needs.

For the akka issue, here is the akka version we include in build.sbt:
"com.typesafe.akka" %% "akka-actor" % "2.2.1"

-Suren


On Thu, Oct 16, 2014 at 12:23 PM, Mohammed Guller <mohammed@glassbeam.com>
wrote:

>  Daniel,
>
> Thanks for sharing this. It is very helpful.
>
>
>
> The reason I want to use Spark submit is that it provides more
> flexibility. For example, with spark-submit, I don’t need to hard code the
> master info in the code. I can easily change the config without having to
> change and recompile code.
>
>
>
> Do you mind sharing the sbt build file for your play app? I tried to build
> an uber jar using sbt-assembly. It gets built, but when I run it, it throws
> all sorts of exception. I have seen some blog posts that Spark and Play use
> different version of the Akka library. So I included Akka in my build.scala
> file, but still cannot get rid of Akka related exceptions. I suspect that
> the settings in the build.scala file for my play project is incorrect.
>
>
>
> Mohammed
>
>
>
> *From:* Daniel Siegmann [mailto:daniel.siegmann@velos.io]
> *Sent:* Thursday, October 16, 2014 7:15 AM
> *To:* Mohammed Guller
> *Cc:* user@spark.apache.org
> *Subject:* Re: Play framework
>
>
>
> We execute Spark jobs from a Play application but we don't use
> spark-submit. I don't know if you really want to use spark-submit, but if
> not you can just create a SparkContext programmatically in your app.
>
> In development I typically run Spark locally. Creating the Spark context
> is pretty trivial:
>
> val conf = new SparkConf().setMaster("local[*]").setAppName(s"My Awesome
> App")
>
> // call conf.set for any other configuration you want
>
> val sc = new SparkContext(sparkConf)
>
> It is important to keep in mind you cannot have multiple local contexts
> (you can create them but you'll get odd errors), so if you are running
> things in parallel within your app (even unit tests) you'd need to share a
> context in this case. If you are running sequentially you can create a new
> local context each time, but you must make sure to call SparkContext.stop()
> when you're done.
>
> Running against a cluster is a bit more complicated because you need to
> add all your dependency jars. I'm not sure how to get this to work with play
> run. I stick to building the app with play dist and then running against
> the packaged application, because it very conveniently provides all the
> dependencies in a lib folder. Here is some code to load all the paths you
> need from the dist:
>
>
>     def libs : Seq[String] = {
>         val libDir = play.api.Play.application.getFile("lib")
>
>         logger.info(s"SparkContext will be initialized with libraries
> from directory $libDir")
>
>         return if ( libDir.exists ) {
>
> libDir.listFiles().map(_.getCanonicalFile().getAbsolutePath()).filter(_.endsWith(".jar"))
>         } else {
>             throw new IllegalStateException(s"lib dir is missing: $libDir")
>         }
>     }
>
> Creating the context is similar to above, but with this extra line:
>
>
> conf.setJars(libs)
>
> I hope this helps. I should note that I don't use play run very much, at
> least not for when I'm actually executing Spark jobs. So I'm not sure if
> this integrates properly with that. I have unit tests which execute on
> Spark and have executed the dist package both locally and on a cluster. To
> make working with the dist locally easier, I wrote myself a little shell
> script to unzip and run the dist.
>
>
>
>
>
> On Wed, Oct 15, 2014 at 10:51 PM, Mohammed Guller <mohammed@glassbeam.com>
> wrote:
>
> Hi –
>
>
>
> Has anybody figured out how to integrate a Play application with Spark and
> run it on a Spark cluster using spark-submit script? I have seen some blogs
> about creating a simple Play app and running it locally on a dev machine
> with sbt run command. However, those steps don’t work for Spark-submit.
>
>
>
> If you have figured out how to build and run a Play app with Spark-submit,
> I would appreciate if you could share the steps and the sbt settings for
> your Play app.
>
>
>
> Thanks,
>
> Mohammed
>
>
>
>
>
>
> --
>
> Daniel Siegmann, Software Developer
> Velos
>
> Accelerating Machine Learning
>
>
> 440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
> E: daniel.siegmann@velos.io W: www.velos.io
>



-- 

SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <suren.hiraman@sociocast.com>elos.io
W: www.velos.io

Mime
View raw message