spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jschindler <john.schind...@utexas.edu>
Subject Re: Using data in RDD to specify HDFS directory to write to
Date Tue, 18 Nov 2014 00:50:15 GMT
Yes, thank you for suggestion.  The error I found below was in the worker
logs.

AssociationError [akka.tcp://sparkWorker@cloudera01.local.company.com:7078]
-> [akka.tcp://sparkExecutor@cloudera01.local.company.com:33329]: Error
[Association failed with
[akka.tcp://sparkExecutor@cloudera01.local.company.com:33329]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkExecutor@cloudera01.local.company.com:33329]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: cloudera01.local.company.com/10.40.19.67:33329
]

I looked into suggestions for this type of error and before I found out the
real reason for the error I upgraded my CDH to 5.2 so I could try setting
the driver and executor ports rather than have Spark choose them at random. 
My boss later turned off iptables and I no longer get that error. I do get a
different one however.  I have gone back into my project and changed my
hadoop version to 2.5.0-cdh5.2.0 so that should not be a problem.

from the master logs

2014-11-17 18:09:49,707 ERROR akka.remote.EndpointWriter: AssociationError
[akka.tcp://sparkMaster@cloudera01.local.local.com:7077] ->
[akka.tcp://spark@localhost:38181]: Error [Association failed with
[akka.tcp://spark@localhost:38181]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://spark@localhost:38181]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: localhost/127.0.0.1:38181
]

2014-11-17 18:19:08,271 INFO akka.actor.LocalActorRef: Message
[akka.remote.transport.AssociationHandle$Disassociated] from
Actor[akka://sparkMaster/deadLetters] to
Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.40.19.67%3A37795-29#-1248895472]
was not delivered. [30] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
2014-11-17 18:19:28,251 ERROR Remoting:
org.apache.spark.deploy.ApplicationDescription; local class incompatible:
stream classdesc serialVersionUID = 583745679236071411, local class
serialVersionUID = 7674242335164700840
java.io.InvalidClassException:
org.apache.spark.deploy.ApplicationDescription; local class incompatible:
stream classdesc serialVersionUID = 583745679236071411, local class
serialVersionUID = 7674242335164700840
        at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
        at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
        at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
        at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at
akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
        at
akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
        at
akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
        at scala.util.Try$.apply(Try.scala:161)
        at
akka.serialization.Serialization.deserialize(Serialization.scala:98)
        at
akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:58)
        at
akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
        at scala.util.Try$.apply(Try.scala:161)
        at
akka.serialization.Serialization.deserialize(Serialization.scala:98)
        at
akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
        at
akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:55)
        at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:55)
        at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:73)
        at
akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:764)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
        at akka.actor.ActorCell.invoke(ActorCell.scala:456)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
        at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

and from regular output

14/11/17 18:09:49 ERROR SparkDeploySchedulerBackend: Application has been
killed. Reason: All masters are unresponsive! Giving up.
14/11/17 18:09:49 ERROR TaskSchedulerImpl: Exiting due to error from cluster
scheduler: All masters are unresponsive! Giving up.

Seems pretty apparent the masters are unresponsive but why is the question?
It is the inverse of the first problem I was having where the executors were
unresponsive.  













--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-data-in-RDD-to-specify-HDFS-directory-to-write-to-tp18789p19114.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message