spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krishnanand Khambadkone <kkhambadk...@yahoo.com.INVALID>
Subject Re: Spark Error: Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
Date Tue, 03 Mar 2015 16:55:45 GMT
Hello Ted,  Some progress,  now it seems that the spark job does get submitted,  In the
spark web UI, I do see this under finished drivers.  However, it seems to not go past this
step,  JavaPairReceiverInputDStream<String, String> messages = KafkaUtils.createStream(jsc,
"localhost:2181", "aa", topicMap);   I do have several messages in the Kafka topic.  When
I run the consumer with from-beginning option, I do see all the messages.

krishs-mbp:bin hadoop$ ./spark-submit --jars /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar
--class KafkaMain --master spark://krishs-mbp:7077 --deploy-mode cluster --num-executors 100
--driver-memory 1g --executor-memory 1g --supervise file:///Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar
Sending launch command to spark://krishs-mbp:7077
Driver successfully submitted as driver-20150303085115-0007
... waiting before polling master for driver state
... polling master for driver state
State of driver-20150303085115-0007 is FINISHED
 

     On Monday, March 2, 2015 4:31 PM, Ted Yu <yuzhihong@gmail.com> wrote:
   

 In AkkaUtils.scala:

    val akkaLogLifecycleEvents = conf.getBoolean("spark.akka.logLifecycleEvents", false)

Can you turn on life cycle event logging to see if you would get some more clue ?
Cheers
On Mon, Mar 2, 2015 at 3:56 PM, Krishnanand Khambadkone <kkhambadkone@yahoo.com> wrote:

I see these messages now,
spark.master -> spark://krishs-mbp:7077
Classpath elements:



Sending launch command to spark://krishs-mbp:7077
Driver successfully submitted as driver-20150302155433-0000
... waiting before polling master for driver state
... polling master for driver state
State of driver-20150302155433-0000 is FAILED
 

     On Monday, March 2, 2015 2:42 PM, Ted Yu <yuzhihong@gmail.com> wrote:
   

 bq. Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
There should be some more output following the above line.
Can you post them ?
Cheers
On Mon, Mar 2, 2015 at 2:06 PM, Krishnanand Khambadkone <kkhambadkone@yahoo.com.invalid>
wrote:

Hi,  I am running spark on my mac.   It is reading from a kafka topic and then writes the
data to a hbase table.  When I do a spark submit,  I get this error,
Error connecting to master spark://localhost:7077 (akka.tcp://sparkMaster@localhost:7077),
exiting.
Cause was: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
My submit statement looks like this,
./spark-submit --jars /Users/hadoop/kafka-0.8.1.1-src/core/build/libs/kafka_2.8.0-0.8.1.1.jar
--class KafkaMain --master spark://localhost:7077 --deploy-mode cluster --num-executors 100
--driver-memory 1g --executor-memory 1g /Users/hadoop/dev/kafkahbase/kafka/src/sparkhbase.jar





    



   
Mime
View raw message