spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Suryavansh <suryavanshi.m...@gmail.com>
Subject Re: Play framework
Date Thu, 16 Oct 2014 22:31:44 GMT
Hi,

Below is the link for a simple Play + SparkSQL example -

http://blog.knoldus.com/2014/07/14/play-with-spark-building-apache-spark-with-play-framework-part-3/

https://github.com/knoldus/Play-Spark-Scala

Manu



On Thu, Oct 16, 2014 at 1:00 PM, Mohammed Guller <mohammed@glassbeam.com>
wrote:

>  Thanks, Suren and Raju.
>
>
>
> Raju – if I remember correctly, Play package command just creates a jar
> for your app. That jar file will not include other dependencies. So it is
> not really a full jar as you mentioned below. So how you are passing all
> the other dependency jars to spark? Can you share that piece of code? Also
> is there any specific reason why you are not using play dist instead?
>
>
>
> Mohammed
>
>
>
> *From:* US Office Admin [mailto:admin@Vectorum.com]
> *Sent:* Thursday, October 16, 2014 11:41 AM
> *To:* Surendranauth Hiraman; Mohammed Guller
>
> *Cc:* Daniel Siegmann; user@spark.apache.org
> *Subject:* Re: Play framework
>
>
>
> ​We integrated Spark into Play and use SparkSQL extensively on an ec2
> spark cluster on Hadoop hdfs 1.2.1 and tachyon 0.4.
>
>
>
> Step 1: Create a play scala application as usual
>
> Step 2. In Build.sbt put all your spark dependencies. What works for us is
> Play 2.2.3 Scala 2.10.4 Spark 1.1. We have Akka 2.2.3. This is straight
> forward
>
> step3: As Daniel mentioned, create spark context within Play. And rest of
> the application is as usual.
>
> Step4: Create a full jar using Play Package and use that package to be
> included in library of jars passed to spark context.
>
> Step 5: Play run as usual.
>
>
>
> It works very well, and the convenience is, we have all scala application
> throughout.
>
>
>
> Regards
>
> Raju
>
>
>
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>    ------------------------------
>
> *From:* Surendranauth Hiraman <suren.hiraman@velos.io>
> *Sent:* Thursday, October 16, 2014 12:42 PM
> *To:* Mohammed Guller
> *Cc:* Daniel Siegmann; user@spark.apache.org
> *Subject:* Re: Play framework
>
>
>
> Mohammed,
>
>
>
> Jumping in for Daniel, we actually address the configuration issue by
> pulling values from environment variables or command line options. Maybe
> that may handle at least some of your needs.
>
>
>
> For the akka issue, here is the akka version we include in build.sbt:
> "com.typesafe.akka" %% "akka-actor" % "2.2.1"
>
>
>
> -Suren
>
>
>
>
>
> On Thu, Oct 16, 2014 at 12:23 PM, Mohammed Guller <mohammed@glassbeam.com>
> wrote:
>
> Daniel,
>
> Thanks for sharing this. It is very helpful.
>
>
>
> The reason I want to use Spark submit is that it provides more
> flexibility. For example, with spark-submit, I don’t need to hard code the
> master info in the code. I can easily change the config without having to
> change and recompile code.
>
>
>
> Do you mind sharing the sbt build file for your play app? I tried to build
> an uber jar using sbt-assembly. It gets built, but when I run it, it throws
> all sorts of exception. I have seen some blog posts that Spark and Play use
> different version of the Akka library. So I included Akka in my build.scala
> file, but still cannot get rid of Akka related exceptions. I suspect that
> the settings in the build.scala file for my play project is incorrect.
>
>
>
> Mohammed
>
>
>
> *From:* Daniel Siegmann [mailto:daniel.siegmann@velos.io]
> *Sent:* Thursday, October 16, 2014 7:15 AM
> *To:* Mohammed Guller
> *Cc:* user@spark.apache.org
> *Subject:* Re: Play framework
>
>
>
> We execute Spark jobs from a Play application but we don't use
> spark-submit. I don't know if you really want to use spark-submit, but if
> not you can just create a SparkContext programmatically in your app.
>
> In development I typically run Spark locally. Creating the Spark context
> is pretty trivial:
>
> val conf = new SparkConf().setMaster("local[*]").setAppName(s"My Awesome
> App")
>
> // call conf.set for any other configuration you want
>
> val sc = new SparkContext(sparkConf)
>
> It is important to keep in mind you cannot have multiple local contexts
> (you can create them but you'll get odd errors), so if you are running
> things in parallel within your app (even unit tests) you'd need to share a
> context in this case. If you are running sequentially you can create a new
> local context each time, but you must make sure to call SparkContext.stop()
> when you're done.
>
> Running against a cluster is a bit more complicated because you need to
> add all your dependency jars. I'm not sure how to get this to work with play
> run. I stick to building the app with play dist and then running against
> the packaged application, because it very conveniently provides all the
> dependencies in a lib folder. Here is some code to load all the paths you
> need from the dist:
>
>
>     def libs : Seq[String] = {
>         val libDir = play.api.Play.application.getFile("lib")
>
>         logger.info(s"SparkContext will be initialized with libraries
> from directory $libDir")
>
>         return if ( libDir.exists ) {
>
> libDir.listFiles().map(_.getCanonicalFile().getAbsolutePath()).filter(_.endsWith(".jar"))
>         } else {
>             throw new IllegalStateException(s"lib dir is missing: $libDir")
>         }
>     }
>
> Creating the context is similar to above, but with this extra line:
>
>
> conf.setJars(libs)
>
> I hope this helps. I should note that I don't use play run very much, at
> least not for when I'm actually executing Spark jobs. So I'm not sure if
> this integrates properly with that. I have unit tests which execute on
> Spark and have executed the dist package both locally and on a cluster. To
> make working with the dist locally easier, I wrote myself a little shell
> script to unzip and run the dist.
>
>
>
>
>
> On Wed, Oct 15, 2014 at 10:51 PM, Mohammed Guller <mohammed@glassbeam.com>
> wrote:
>
> Hi –
>
>
>
> Has anybody figured out how to integrate a Play application with Spark and
> run it on a Spark cluster using spark-submit script? I have seen some blogs
> about creating a simple Play app and running it locally on a dev machine
> with sbt run command. However, those steps don’t work for Spark-submit.
>
>
>
> If you have figured out how to build and run a Play app with Spark-submit,
> I would appreciate if you could share the steps and the sbt settings for
> your Play app.
>
>
>
> Thanks,
>
> Mohammed
>
>
>
>
>
>
> --
>
> Daniel Siegmann, Software Developer
> Velos
>
> Accelerating Machine Learning
>
>
> 440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
> E: daniel.siegmann@velos.io W: www.velos.io
>
>
>
>
>
> --
>
>
>
> SUREN HIRAMAN, VP TECHNOLOGY
> Velos
>
> Accelerating Machine Learning
>
> 440 NINTH AVENUE, 11TH FLOOR
> NEW YORK, NY 10001
> O: (917) 525-2466 ext. 105
>
> F: 646.349.4063
> E: suren.hiraman@v <suren.hiraman@sociocast.com>elos.io
> W: www.velos.io
>



-- 
Manu Suryavansh

Mime
View raw message