spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: sbt package failed: wrong libraryDependencies for spark-streaming?
Date Fri, 01 Aug 2014 01:12:34 GMT
I dont see the error.

The twitter stuff (as well as kafka and flume stuff) are treated as
"external" projects and are not included in the spark shell. This is
because we dont want the dependencies of such non-core functionalities to
cause random conflicts with that of core spark.

Hence its not possible to use twitter spark streaming in the spark shell.
Unless you explicitly  add the jar generated in
external/twitter/target/scala... To the spark shells extra class path (see
spark config page)

TD

On Thu, Jul 31, 2014 at 5:48 PM, durin <mail@simon-schaefer.net> wrote:
> As suggested here
> <
http://apache-spark-user-list.1001560.n3.nabble.com/import-org-apache-spark-streaming-twitter-in-Shell-tp9665p9743.html
>
> , I want to create a minimal project using sbt to be able to use
> org.apache.spark.streaming.twitter in the shell. My Spark version is the
> latest Master branch compiled with maven3.
>
> I was following this
> <
http://spark.apache.org/docs/latest/quick-start.html#standalone-applications
>
> tutorial to create a small standalone application.
>
> My simple.sbt file looks like this:
>
>
>
>
> However, I will get this error when using sbt package:
>
>
>
>
> I don't quite understand the whole standalone thing yet. sbt is installed
on
> a different machine than my Spark Cluster, but the tutorial never mentions
> any variables that would have to be set. So I did assume sbt didn't
actually
> need a Spark installation on the machine.
> Also, I could imagine that 1.0.1 isn't the correct version when I compiled
> it myself.
>
>
> Can someone help me out? In the end, all I want for now is to be able to
use
> twitterUtils in the spark-shell.
>
>
> Simon
>
>
>
> --
> View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message