spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harold Nguyen <har...@nexgate.com>
Subject Re: Including jars in Spark-shell vs Spark-submit
Date Tue, 28 Oct 2014 19:03:18 GMT
Hi Helena,

It's great to e-meet you! I've actually been following along your blogs and
talks trying to get this to work.

I just solved it, and you were absolutely correct. I've been using
1.1.0-alpha3 as my dependency, but my assembly is the 1.2.0-SNAPSHOT.

Thanks for looking through all my other threads to piece together the
problem.

So the solution is that I just included the
spark-cassandra-connector-assembly-1.2.0-SNAPSHOT
straight into a "lib" folder, removed the line from 1.1.0-alpha3 line from
libraryDepenencies, and re-packaged it.

It all works great now, and I couldn't be happier. I had to piece together
6 different forums and sites to get that working (being absolutely new to
Spark and Scala and sbt). I'll write a blog post on how to get this working
later, in case it can help someone.

 I really appreciate the help!

Harold

On Tue, Oct 28, 2014 at 11:55 AM, Helena Edelson <
helena.edelson@datastax.com> wrote:

> Hi Harold,
> It seems like, based on your previous post, you are using one version of
> the connector as a dependency yet building the assembly jar from master?
> You were using 1.1.0-alpha3 (you can upgrade to alpha4, beta coming this
> week) yet your assembly is
> spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.
>
> libraryDependencies ++= Seq(
>   "org.apache.spark" % "spark-streaming_2.10" % "1.1.0",
>   "com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0-alpha3"
> withSources() withJavadoc(),
>   "org.apache.spark" %% "spark-sql" % "1.1.0"
> )
>
> - Helena
>
> On Oct 28, 2014, at 2:08 PM, Harold Nguyen <harold@nexgate.com> wrote:
>
> > Hi all,
> >
> > The following works fine when submitting dependency jars through
> Spark-Shell:
> >
> > ./bin/spark-shell --master spark://ip-172-31-38-112:7077 --jars
> /home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
> >
> > But not through spark-submit:
> >
> >  ./bin/spark-submit --class
> "org.apache.spark.examples.streaming.CassandraSave" --master
> spark://ip-172-31-38-112:7077
> streaming-test/target/scala-2.10/simple-streaming_2.10-1.0.jar --jars
> local:///home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
> >
> > Am I issuing the spark-submit command incorrectly ? Each of the workers
> has that built jar in their respective directories
> (spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar)
> >
> > Thanks,
> >
> > Harold
>
>

Mime
View raw message