spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shafaq <s.abdullah...@gmail.com>
Subject Re: New sbt plugin to deploy jobs to EC2
Date Wed, 24 Sep 2014 19:35:55 GMT
Hi,

  testing out the Spark Ec2 deployment plugin:

I try to compile using

 $sbt sparkLaunchCluster

-----------------------------------------------------------------------------------------------
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn]     ::::::::::::::::::::::::::::::::::::::::::::::
[warn]     ::          UNRESOLVED DEPENDENCIES         ::
[warn]     ::::::::::::::::::::::::::::::::::::::::::::::
[warn]     :: fr.janalyse#janalyse-ssh_2.10;0.9.13: not found
[warn]     ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn]     Note: Unresolved dependencies path:
[warn]         fr.janalyse:janalyse-ssh_2.10:0.9.13
[warn]           +- com.gilt:lib-spark-manager_2.10:0.0.3.9
[warn]           +- com.gilt:sbt-spark-ec2-plugin:0.1.5 (sbtVersion=0.13,
scalaVersion=2.10)
(/Users/saq/Work/spark-datapipeline/project/plugins.sbt#L6-7)
[warn]           +- default:spark-datapipeline-build:0.1-SNAPSHOT
(sbtVersion=0.13, scalaVersion=2.10)
sbt.ResolveException: unresolved dependency:
fr.janalyse#janalyse-ssh_2.10;0.9.13: not found
    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:243)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:158)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:156)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
    at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:124)
    at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:56)
    at sbt.IvySbt$$anon$3.call(Ivy.scala:64)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
    at
xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
    at
xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
    at xsbt.boot.Using$.withResource(Using.scala:10)
    at xsbt.boot.Using$.apply(Using.scala:9)
    at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
    at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
    at xsbt.boot.Locks$.apply0(Locks.scala:31)
    at xsbt.boot.Locks$.apply(Locks.scala:28)
    at sbt.IvySbt.withDefaultLogger(Ivy.scala:64)
    at sbt.IvySbt.withIvy(Ivy.scala:119)
    at sbt.IvySbt.withIvy(Ivy.scala:116)
    at sbt.IvySbt$Module.withModule(Ivy.scala:147)

------------------------------------------------------

I have to use scala version 2.10 as spark is using it and sbt version is
0.13

My sbt looks as follows:

name := "scala-datapipeline"

version := "1.0"

scalaVersion := "2.10.4"


scalacOptions ++= Seq( "-deprecation", "-unchecked", "-feature")



sparkec2.Ec2SparkPluginSettings.sparkSettings


resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

resolvers += "spray" at "http://repo.spray.io/"


resolvers += Resolver.url(
  "bintray Repository",
  url("http://dl.bintray.com/felixgborrego/repo"))(
    Resolver.ivyStylePatterns)

resolvers += "JAnalyse Repository" at "http://www.janalyse.fr/repository/"





libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "1.1.0",    // 1.0.2
    "org.apache.spark" %% "spark-sql"  % "1.1.0",
    "org.apache.spark" %% "spark-hive"  % "1.1.0",
    "com.github.nscala-time" %% "nscala-time" % "1.0.0",
    "org.json4s" %% "json4s-native" % "3.2.10",
    "com.codahale" %% "jerkson_2.9.1" % "0.5.0",
    "fr.janalyse" % "janalyse-ssh" % "0.9.10"

)










On Fri, Sep 5, 2014 at 4:08 AM, andy petrella <andy.petrella@gmail.com>
wrote:

> \o/ => will test it soon or sooner, gr8 idea btw
>
> aℕdy ℙetrella
> about.me/noootsab
> [image: aℕdy ℙetrella on about.me]
>
> <http://about.me/noootsab>
>
>
> On Fri, Sep 5, 2014 at 12:37 PM, Felix Garcia Borrego <fborrego@gilt.com>
> wrote:
>
>> As far as I know in other to deploy and execute jobs in EC2 you need to
>> assembly you project, copy your jar into the cluster, log into using ssh
>> and submit the job.
>>
>> To avoid having to do this I've been prototyping an sbt plugin(1) that
>> allows to create and send Spark jobs to an Amazon EC2 cluster directly from
>> your local machine using sbt.
>>
>> It's a simple plugin that actually rely on spark-ec2 and spark-submit,
>> but  I'd like to have feedback and see if this plugin makes any sense
>> before going ahead with the final impl or if there is any other easy way to
>> do so.
>>
>> (1) https://github.com/felixgborrego/sbt-spark-ec2-plugin
>>
>> Thanks,
>>
>>
>>
>


-- 
Kind Regards,
Shafaq

Mime
View raw message