spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammed Guller <moham...@glassbeam.com>
Subject RE: Spark and Play
Date Thu, 13 Nov 2014 17:22:07 GMT
Hi Patrick,
Although we are able to use Spark 1.1.0 with Play 2.2.x, as you mentioned, Akka incompatibility
prevents us from using Spark with the current stable releases of Play (2.3.6) and Akka (2.3.7).
Are there any plans to address this issue in Spark 1.2?

Thanks,
Mohammed

From: John Meehan [mailto:jnmeehan@gmail.com]
Sent: Tuesday, November 11, 2014 11:35 PM
To: Mohammed Guller
Cc: Patrick Wendell; Akshat Aranya; user@spark.apache.org
Subject: Re: Spark and Play

You can also build a Play 2.2.x + Spark 1.1.0 fat jar with sbt-assembly for, e.g. yarn-client
support or using with spark-shell for debugging:

play.Project.playScalaSettings

libraryDependencies ~= { _ map {
  case m if m.organization == "com.typesafe.play" =>
    m.exclude("commons-logging", "commons-logging")
  case m => m
}}

assemblySettings

test in assembly := {}

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
    case m if m.startsWith("META-INF") => MergeStrategy.discard
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first
    case PathList("org", "apache", xs @ _*) => MergeStrategy.first
    case PathList("org", "jboss", xs @ _*) => MergeStrategy.first
    case PathList("org", "slf4j", xs @ _*) => MergeStrategy.discard
    case "about.html"  => MergeStrategy.rename
    case "reference.conf" => MergeStrategy.concat
    case _ => MergeStrategy.first
  }
}

On Tue, Nov 11, 2014 at 3:04 PM, Mohammed Guller <mohammed@glassbeam.com<mailto:mohammed@glassbeam.com>>
wrote:
Actually, it is possible to integrate Spark 1.1.0 with Play 2.2.x

Here is a sample build.sbt file:

name := """xyz"""

version := "0.1 "

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
  jdbc,
  anorm,
  cache,
  "org.apache.spark" %% "spark-core" % "1.1.0",
  "com.typesafe.akka" %% "akka-actor" % "2.2.3",
  "com.typesafe.akka" %% "akka-slf4j" % "2.2.3",
  "org.apache.spark" %% "spark-sql" % "1.1.0"
)

play.Project.playScalaSettings


Mohammed

-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com<mailto:pwendell@gmail.com>]
Sent: Tuesday, November 11, 2014 2:06 PM
To: Akshat Aranya
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Spark and Play

Hi There,

Because Akka versions are not binary compatible with one another, it might not be possible
to integrate Play with Spark 1.1.0.

- Patrick

On Tue, Nov 11, 2014 at 8:21 AM, Akshat Aranya <aaranya@gmail.com<mailto:aaranya@gmail.com>>
wrote:
> Hi,
>
> Sorry if this has been asked before; I didn't find a satisfactory
> answer when searching.  How can I integrate a Play application with
> Spark?  I'm getting into issues of akka-actor versions.  Play 2.2.x
> uses akka-actor 2.0, whereas Play 2.3.x uses akka-actor 2.3.4, neither
> of which work fine with Spark 1.1.0.  Is there something I should do
> with libraryDependencies in my build.sbt to make it work?
>
> Thanks,
> Akshat

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>

Mime
View raw message