spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aecc <alessandroa...@gmail.com>
Subject Kafka in Yarn
Date Thu, 13 Mar 2014 17:58:31 GMT
Hi, I would like to know how is the correct way to add kafka to my project in
StandAlone YARN, given that now it's in a different artifact than the Spark
core.

I tried adding the dependency to my project but I get a
NotClassFoundException to my main class. Also, that makes my Jar file very
big, so it's not convenient.

The next code is how I run it:

SPARK_JAR=/opt/spark/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar
/opt/spark/bin/spark-class org.apache.spark.deploy.yarn.Client --jar
project.jar --class StarterClass --args yarn-standalone  --num-workers 1
--worker-cores 1 --master-memory 1536m --worker-memory 1536m

Thanks



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-in-Yarn-tp2673.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message