spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Helena Edelson <helena.edel...@datastax.com>
Subject Re: Including jars in Spark-shell vs Spark-submit
Date Tue, 28 Oct 2014 18:55:58 GMT
Hi Harold,
It seems like, based on your previous post, you are using one version of the connector as
a dependency yet building the assembly jar from master? You were using 1.1.0-alpha3 (you can
upgrade to alpha4, beta coming this week) yet your assembly is spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-streaming_2.10" % "1.1.0",
  "com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0-alpha3" withSources() withJavadoc(),
  "org.apache.spark" %% "spark-sql" % "1.1.0"
)

- Helena

On Oct 28, 2014, at 2:08 PM, Harold Nguyen <harold@nexgate.com> wrote:

> Hi all,
> 
> The following works fine when submitting dependency jars through Spark-Shell:
> 
> ./bin/spark-shell --master spark://ip-172-31-38-112:7077 --jars /home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
> 
> But not through spark-submit:
> 
>  ./bin/spark-submit --class "org.apache.spark.examples.streaming.CassandraSave" --master
spark://ip-172-31-38-112:7077 streaming-test/target/scala-2.10/simple-streaming_2.10-1.0.jar
--jars local:///home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
> 
> Am I issuing the spark-submit command incorrectly ? Each of the workers has that built
jar in their respective directories (spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar)
> 
> Thanks,
> 
> Harold


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message