spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Berman <igor.ber...@gmail.com>
Subject Re: All masters are unresponsive! Giving up.
Date Fri, 07 Aug 2015 18:08:34 GMT
check on which ip/port master listens
netstat -a -t --numeric-ports


On 7 August 2015 at 20:48, Jeff Jones <jjones@adaptivebiotech.com> wrote:

> Thanks. Added this to both the client and the master but still not getting
> any more information. I confirmed the flag with ps.
>
>
>
> jjones    53222  2.7  0.1 19399412 549656 pts/3 Sl   17:17   0:44
> /opt/jdk1.8/bin/java -cp
> /home/jjones/bin/spark-1.4.1-bin-hadoop2.6/sbin/../conf/:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar
> -Dsun.io.serialization.extendedDebugInfo=true -Xms512m -Xmx512m
> org.apache.spark.deploy.master.Master --ip p3.ourdomain.com --port 7077
> --webui-port 8080’
>
>
>
> Error message(s) the same:
>
>
>
> 15/08/07 17:23:26 ERROR Remoting: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> java.io.InvalidClassException: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
>         at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
>
>         at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
>
>         at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
>
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
>         at
> akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
>
>         at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>
>         at
> akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
>
>         at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
>         at scala.util.Try$.apply(Try.scala:161)
>
>         at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
>         at
> akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
>
>         at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
>         at scala.util.Try$.apply(Try.scala:161)
>
>         at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
>         at
> akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
>
>         at
> akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
>
>         at
> akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
>
>         at
> akka.remote.DefaultMessageDispatcher.payloadClass$1(Endpoint.scala:59)
>
>         at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:99)
>
>         at
> akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
>
>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>
>         at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
>
>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>
>         at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>
>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>
>         at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>
>         at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>
>
>
> *From:* Sonal Goyal [mailto:sonalgoyal4@gmail.com]
> *Sent:* Thursday, August 6, 2015 11:22 PM
> *To:* Jeff Jones
> *Cc:* user@spark.apache.org
> *Subject:* Re: All masters are unresponsive! Giving up.
>
>
>
> There seems  to be a version mismatch somewhere. You can try and find out
> the cause with debug serialization information. I think the jvm flag
> -Dsun.io.*serialization*.*extendedDebugInfo*=true should help.
>
>
> Best Regards,
> Sonal
> Founder, Nube Technologies <http://www.nubetech.co>
>
> Check out Reifier at Spark Summit 2015
> <https://spark-summit.org/2015/events/real-time-fuzzy-matching-with-spark-and-elastic-search/>
>
>
>
>
>
> On Fri, Aug 7, 2015 at 4:42 AM, Jeff Jones <jjones@adaptivebiotech.com>
> wrote:
>
> I wrote a very simple Spark 1.4.1 app that I can run through a local
> driver program just fine using setMaster(“local[*]”).  The app is as
> follows:
>
>
>
> import org.apache.spark.SparkContext
>
> import org.apache.spark.SparkContext._
>
> import org.apache.spark.SparkConf
>
> import org.apache.spark.rdd.RDD
>
>
>
> object Hello {
>
>   def main(args: Array[String]): Unit = {
>
>     val conf = new SparkConf().setAppName("Simple
> Application").setMaster("local[*]")
>
>     val sc = new SparkContext(conf)
>
>     val data:RDD[Int] = sc.parallelize(Seq(1,2,12,34,2354,123,100), 2)
>
>     println("Max: " + data.max)
>
>     println("Min: " + data.min)
>
>   }
>
> }
>
>
>
> I compile this using the following build.sbt which will pull the needed
> Spark libraries for me.
>
>
>
> name := """SparkyJeff"""
>
>
>
> version := "1.0"
>
>
>
> scalaVersion := "2.11.6"
>
>
>
> // Change this to another test framework if you prefer
>
> libraryDependencies ++= Seq(
>
>         "org.apache.spark" %% "spark-core" % "1.4.1",
>
>         "org.apache.spark" %% "spark-sql"  % "1.4.1")
>
>
>
> // Uncomment to use Akka
>
> //libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.11"
>
>
>
> fork := true
>
>
>
> Now I’m trying to run this against a standalone cluster by changing the
> setMaster(“local[*]”) to setMaster(“spark://p3.ourdomain.com:7077”). I
> downloaded Spark 1.4.1 for Hadoop 2.6 or greater.  Set the SPARK_MASTER_IP=”
> p3.ourdomain.com”, SPARK_WORKER_CORES=”1000”,SPARK_WORKER_MEMORY=”500g”
> and then started the cluster using run-all.sh. The cluster appears to start
> fine. I can hit cluster UI at p3.ourdomain.com:8080 and see the same
> master URL as mentioned above.
>
>
>
> Now when I run my little app I get the following client error:
>
>
>
> …
>
> [error] 15/08/05 16:03:40 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:03:40 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:00 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:04:00 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:20 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:04:20 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:40 ERROR SparkDeploySchedulerBackend: Application
> has been killed. Reason: All masters are unresponsive! Giving up.
>
> …
>
>
>
> Looking into the master logs I find:
>
>
>
> 15/08/06 22:52:28 INFO Master: akka.tcp://sparkDriver@192.168.137.41:48877
> got disassociated, removing it.
>
> 15/08/06 22:52:46 ERROR Remoting: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> java.io.InvalidClassException: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
>         at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
>
>         at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
>
>         at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
>
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
>         at
> akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
>
>         at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>
>         at
> akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
>
>         at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
>         at scala.util.Try$.apply(Try.scala:161)
>
>         at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
>         at
> akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
>
>         at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
>         at scala.util.Try$.apply(Try.scala:161)
>
>         at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
>         at
> akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
>
>         at
> akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
>
>         at
> akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
>
>         at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
>
>         at
> akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
>
>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>
>         at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
>
>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>
>         at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>
>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>
>         at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>
>         at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>
>         at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
>         at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>
>         at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
>         at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
>
> Any help would be greatly appreciated.
>
>
>
> Thanks,
> Jeff
>
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>
>
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>

Mime
View raw message