spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject Re: spark 0.8
Date Fri, 18 Oct 2013 21:09:27 GMT
OK it turned out setting
-Dspark.serializer=org.apache.spark.serializer.KryoSerializer in
SPARK_JAVA_OPTS on the workers/slaves caused all this. not sure why. this
used to work fine in previous spark. but when i removed it the errors went
away.


On Fri, Oct 18, 2013 at 2:59 PM, Koert Kuipers <koert@tresata.com> wrote:

> i installed the plain vanilla spark 0.8 on our cluster, downloaded from
> here:
> http://spark-project.org/download/spark-0.8.0-incubating-bin-hadoop1.tgz
> after a restart of all spark daemons i still see the same issue for every
> task:
>
> java.io.StreamCorruptedException: invalid type code: 00
>
> so now i am guessing it must be something in my configuration. i guess
> this is progress...
>
> looking at the logs of a worker, i see the task gets launched like this:
>
> Spark Executor Command: "java" "-cp"
> ":/usr/local/lib/spark/conf:/usr/local/lib/spark/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.0-incubating-hadoop1.0.4.jar"
> "-Dspar
> k.worker.timeout=30000" "-Dspark.akka.timeout=30000"
> "-Dspark.storage.blockManagerHeartBeatMs=120000"
> "-Dspark.storage.blockManagerTimeoutIntervalMs=120000" "-Dspark.akka.retry
> .wait=30000" "-Dspark.akka.frameSize=10000"
> "-Dspark.akka.logLifecycleEvents=true"
> "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer"
> "-Dspark.worker.timeout=30000
> " "-Dspark.akka.timeout=30000"
> "-Dspark.storage.blockManagerHeartBeatMs=120000"
> "-Dspark.storage.blockManagerTimeoutIntervalMs=120000"
> "-Dspark.akka.retry.wait=30000" "-Dspark.
> akka.frameSize=10000" "-Dspark.akka.logLifecycleEvents=true"
> "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer" "-Xms512M"
> "-Xmx512M" "org.apache.spark.executor.St
> andaloneExecutorBackend" "akka://
> spark@192.168.3.171:38472/user/StandaloneScheduler" "1" "node02" "7"
>
>
> and finally this is my spark-env.sh:
>
> export SCALA_HOME=/usr/local/lib/scala-2.9.3
> export SPARK_MASTER_IP=node01
> export SPARK_MASTER_PORT=7077
> export SPARK_MASTER_WEBUI_PORT=8080
> export SPARK_WORKER_CORES=7
> export SPARK_WORKER_MEMORY=14G
> export SPARK_WORKER_PORT=7078
> export SPARK_WORKER_WEBUI_PORT=8081
> export SPARK_WORKER_DIR=/var/lib/spark
> export SPARK_CLASSPATH=$SPARK_USER_CLASSPATH
> export SPARK_JAVA_OPTS="-Dspark.worker.timeout=30000
> -Dspark.akka.timeout=30000 -Dspark.storage.blockManagerHeartBeatMs=120000
> -Dspark.storage.blockManagerTimeoutIntervalMs=120
> 000 -Dspark.akka.retry.wait=30000 -Dspark.akka.frameSize=10000
> -Dspark.akka.logLifecycleEvents=true
> -Dspark.serializer=org.apache.spark.serializer.KryoSerializer $SPARK_JAVA_OP
> TS"
> export
> SPARK_WORKER_OPTS="-Dspark.local.dir=/data/0/tmp,/data/1/tmp,/data/2/tmp,/data/3/tmp,/data/4/tmp,/data/5/tmp"
>
>
>
>
>
> On Fri, Oct 18, 2013 at 2:02 PM, Koert Kuipers <koert@tresata.com> wrote:
>
>> i checked out the v0.8.0-incubating tag again, changed the settings to
>> build against correct version of hadoop for our cluster, ran sbt-assembly,
>> build tarball, installed it on cluster, restarted spark... same errors
>>
>>
>> On Fri, Oct 18, 2013 at 12:49 PM, Koert Kuipers <koert@tresata.com>wrote:
>>
>>> at this point i feel like it must be some sort of version mismatch? i am
>>> gonna check the spark build that i deployed on the cluster
>>>
>>>
>>> On Fri, Oct 18, 2013 at 12:46 PM, Koert Kuipers <koert@tresata.com>wrote:
>>>
>>>> name := "Simple Project"
>>>>
>>>> version := "1.0"
>>>>
>>>> scalaVersion := "2.9.3"
>>>>
>>>> libraryDependencies += "org.apache.spark" %% "spark-core" %
>>>> "0.8.0-incubating"
>>>>
>>>> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
>>>>
>>>> resolvers += "Cloudera Repository" at "
>>>> https://repository.cloudera.com/artifactory/cloudera-repos/"
>>>>
>>>> libraryDependencies += "org.apache.hadoop" % "hadoop-client" %
>>>> "2.0.0-mr1-cdh4.3.0"
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Oct 18, 2013 at 12:34 PM, Matei Zaharia <
>>>> matei.zaharia@gmail.com> wrote:
>>>>
>>>>> Can you post the build.sbt for your program? It needs to include
>>>>> hadoop-client for CDH4.3, and that should *not* be listed as provided.
>>>>>
>>>>> Matei
>>>>>
>>>>> On Oct 18, 2013, at 8:23 AM, Koert Kuipers <koert@tresata.com>
wrote:
>>>>>
>>>>> ok this has nothing to do with hadoop access. even a simple program
>>>>> that uses sc.parallelize blows up in this way.
>>>>>
>>>>> so spark-shell works well on the same machine i launch this from.
>>>>>
>>>>> if i launch a simple program without using kryo for serializer and
>>>>> closure serialize i get a different error. see below.
>>>>> at this point it seems to me i have some issue with task
>>>>> serialization???
>>>>>
>>>>>
>>>>>
>>>>> 13/10/18 11:20:37 INFO StandaloneExecutorBackend: Got assigned task 0
>>>>> 13/10/18 11:20:37 INFO StandaloneExecutorBackend: Got assigned task 1
>>>>> 13/10/18 11:20:37 INFO Executor: Running task ID 1
>>>>> 13/10/18 11:20:37 INFO Executor: Running task ID 0
>>>>> 13/10/18 11:20:37 INFO Executor: Fetching
>>>>> http://192.168.3.171:41629/jars/simple-project_2.9.3-1.0.jar with
>>>>> timestamp 1382109635095
>>>>> 13/10/18 11:20:37 INFO Utils: Fetching
>>>>> http://192.168.3.171:41629/jars/simple-project_2.9.3-1.0.jar to
>>>>> /tmp/fetchFileTemp378181753997570700.tmp
>>>>> 13/10/18 11:20:37 INFO Executor: Adding
>>>>> file:/var/lib/spark/app-20131018112035-0014/1/./simple-project_2.9.3-1.0.jar
>>>>> to class loader
>>>>> 13/10/18 11:20:37 INFO Executor: caught throwable with stacktrace
>>>>> java.io.StreamCorruptedException: invalid type code: 00
>>>>>     at
>>>>> java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2467)
>>>>>     at
>>>>> java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2502)
>>>>>     at
>>>>> java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2661)
>>>>>     at
>>>>> java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2583)
>>>>>     at java.io.DataInputStream.readFully(DataInputStream.java:178)
>>>>>     at java.io.DataInputStream.readLong(DataInputStream.java:399)
>>>>>     at
>>>>> java.io.ObjectInputStream$BlockDataInputStream.readLong(ObjectInputStream.java:2803)
>>>>>     at java.io.ObjectInputStream.readLong(ObjectInputStream.java:958)
>>>>>     at
>>>>> org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:72)
>>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>     at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>     at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>     at
>>>>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
>>>>>     at
>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1852)
>>>>>     at
>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
>>>>>     at
>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
>>>>>     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>>>>>     at
>>>>> org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:135)
>>>>>     at
>>>>> java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1795)
>>>>>     at
>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1754)
>>>>>     at
>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
>>>>>     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>>>>>     at
>>>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39)
>>>>>     at
>>>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61)
>>>>>     at
>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:153)
>>>>>     at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>>     at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>>     at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 18, 2013 at 10:59 AM, Koert Kuipers <koert@tresata.com>wrote:
>>>>>
>>>>>> i created a tiny sbt project as described here:
>>>>>> apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala<http://spark.incubator.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala>
>>>>>>
>>>>>> it has the correct dependencies: spark-core and the correct
>>>>>> hadoop-client for my platform. i tried both the generic spark-core
>>>>>> dependency and spark-core dependency compiled against my platform.
it runs
>>>>>> fine in local mode, but when i switch to the cluster i still always
get the
>>>>>> following exceptions on tasks:
>>>>>>
>>>>>> 13/10/18 10:25:53 ERROR Executor: Uncaught exception in thread
>>>>>> Thread[pool-5-thread-1,5,main]
>>>>>>
>>>>>> java.lang.NullPointerException
>>>>>>     at
>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
>>>>>>     at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>>>     at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>>>     at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> after adding some additional debugging to Executor i see the cause
is
>>>>>> this:
>>>>>> 13/10/18 10:54:47 INFO Executor: caught throwable with stacktrace
>>>>>> java.lang.NullPointerException
>>>>>>     at
>>>>>> org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$2.apply(Executor.scala:155)
>>>>>>     at
>>>>>> org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$2.apply(Executor.scala:155)
>>>>>>     at org.apache.spark.Logging$class.logInfo(Logging.scala:48)
>>>>>>     at org.apache.spark.executor.Executor.logInfo(Executor.scala:36)
>>>>>>     at
>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:155)
>>>>>>
>>>>>>     at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>>>     at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>>>     at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> so it seems the offending line is:
>>>>>> logInfo("Its epoch is " + task.epoch)
>>>>>>
>>>>>> i am guessing accessing epoch on the task is throwing the NPE. any
>>>>>> ideas?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 17, 2013 at 8:12 PM, Koert Kuipers <koert@tresata.com>wrote:
>>>>>>
>>>>>>> sorry one more related question:
>>>>>>> i compile against a spark build for hadoop 1.0.4, but the actual
>>>>>>> installed version of spark is build against cdh4.3.0-mr1. this
also used to
>>>>>>> work, and i prefer to do this so i compile against a generic
spark build.
>>>>>>> could this be the issue?
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 17, 2013 at 8:06 PM, Koert Kuipers <koert@tresata.com>wrote:
>>>>>>>
>>>>>>>> i have my spark and hadoop related dependencies as "provided"
for
>>>>>>>> my spark job. this used to work with previous versions. are
these now
>>>>>>>> supposed to be compile/runtime/default dependencies?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 17, 2013 at 8:04 PM, Koert Kuipers <koert@tresata.com>wrote:
>>>>>>>>
>>>>>>>>> yes i did that and i can see the correct jars sitting
in
>>>>>>>>> lib_managed
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Oct 17, 2013 at 7:56 PM, Matei Zaharia <
>>>>>>>>> matei.zaharia@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Koert, did you link your Spark job to the right version
of HDFS
>>>>>>>>>> as well? In Spark 0.8, you have to add a Maven dependency
on
>>>>>>>>>> "hadoop-client" for your version of Hadoop. See
>>>>>>>>>> http://spark.incubator.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala
for
>>>>>>>>>> example.
>>>>>>>>>>
>>>>>>>>>> Matei
>>>>>>>>>>
>>>>>>>>>> On Oct 17, 2013, at 4:38 PM, Koert Kuipers <koert@tresata.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> i got the job a little further along by also setting
this:
>>>>>>>>>> System.setProperty("spark.closure.serializer",
>>>>>>>>>> "org.apache.spark.serializer.KryoSerializer")
>>>>>>>>>>
>>>>>>>>>> not sure why i need to... but anyhow, now my workers
start and
>>>>>>>>>> then they blow up on this:
>>>>>>>>>>
>>>>>>>>>> 13/10/17 19:22:57 ERROR Executor: Uncaught exception
in thread
>>>>>>>>>> Thread[pool-5-thread-1,5,main]
>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>     at
>>>>>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
>>>>>>>>>>     at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>>>>>>>     at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>>>>>>>     at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> which is:
>>>>>>>>>>  val metrics = attemptedTask.flatMap(t => t.metrics)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Oct 17, 2013 at 7:30 PM, dachuan <hdc1112@gmail.com>wrote:
>>>>>>>>>>
>>>>>>>>>>> thanks, Mark.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Oct 17, 2013 at 6:36 PM, Mark Hamstra
<
>>>>>>>>>>> mark@clearstorydata.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> SNAPSHOTs are not fixed versions, but are
floating names
>>>>>>>>>>>> associated with whatever is the most recent
code.  So, Spark 0.8.0 is the
>>>>>>>>>>>> current released version of Spark, which
is exactly the same today as it
>>>>>>>>>>>> was yesterday, and will be the same thing
forever.  Spark 0.8.1-SNAPSHOT is
>>>>>>>>>>>> whatever is currently in branch-0.8.  It
changes every time new code is
>>>>>>>>>>>> committed to that branch (which should be
just bug fixes and the few
>>>>>>>>>>>> additional features that we wanted to get
into 0.8.0, but that didn't quite
>>>>>>>>>>>> make it.)  Not too long from now there will
be a release of Spark 0.8.1, at
>>>>>>>>>>>> which time the SNAPSHOT will got to 0.8.2
and 0.8.1 will be forever frozen.
>>>>>>>>>>>>  Meanwhile, the wild new development is taking
place on the master branch,
>>>>>>>>>>>> and whatever is currently in that branch
becomes 0.9.0-SNAPSHOT.  This
>>>>>>>>>>>> could be quite different from day to day,
and there are no guarantees that
>>>>>>>>>>>> things won't be broken in 0.9.0-SNAPSHOT.
 Several months from now there
>>>>>>>>>>>> will be a release of Spark 0.9.0 (unless
the decision is made to bump the
>>>>>>>>>>>> version to 1.0.0), at which point the SNAPSHOT
goes to 0.9.1 and the whole
>>>>>>>>>>>> process advances to the next phase of development.
>>>>>>>>>>>>
>>>>>>>>>>>> The short answer is that releases are stable,
SNAPSHOTs are
>>>>>>>>>>>> not, and SNAPSHOTs that aren't on maintenance
branches can break things.
>>>>>>>>>>>>  You make your choice of which to use and
pay the consequences.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Oct 17, 2013 at 3:18 PM, dachuan
<hdc1112@gmail.com>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> yeah, I mean 0.9.0-SNAPSHOT. I use git
clone and that's what I
>>>>>>>>>>>>> got.. what's the difference? I mean SNAPSHOT
and non-SNAPSHOT.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Oct 17, 2013 at 6:15 PM, Mark
Hamstra <
>>>>>>>>>>>>> mark@clearstorydata.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Of course, you mean 0.9.0-SNAPSHOT.
 There is no Spark 0.9.0,
>>>>>>>>>>>>>> and won't be for several months.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Oct 17, 2013 at 3:11 PM,
dachuan <hdc1112@gmail.com>wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm sorry if this doesn't answer
your question directly, but
>>>>>>>>>>>>>>> I have tried spark 0.9.0 and
hdfs 1.0.4 just now, it works..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Oct 17, 2013 at 6:05
PM, Koert Kuipers <
>>>>>>>>>>>>>>> koert@tresata.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> after upgrading from spark
0.7 to spark 0.8 i can no longer
>>>>>>>>>>>>>>>> access any files on HDFS.
>>>>>>>>>>>>>>>>  i see the error below. any
ideas?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> i am running spark standalone
on a cluster that also has
>>>>>>>>>>>>>>>> CDH4.3.0 and rebuild spark
accordingly. the jars in lib_managed look good
>>>>>>>>>>>>>>>> to me.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> i noticed similar errors
in the mailing list but found no
>>>>>>>>>>>>>>>> suggested solutions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thanks! koert
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 13/10/17 17:43:23 ERROR Executor:
Exception in task ID 0
>>>>>>>>>>>>>>>> java.io.EOFException
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2703)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1008)
>>>>>>>>>>>>>>>> 	at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68)
>>>>>>>>>>>>>>>> 	at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106)
>>>>>>>>>>>>>>>> 	at org.apache.hadoop.io.UTF8.readChars(UTF8.java:258)
>>>>>>>>>>>>>>>> 	at org.apache.hadoop.io.UTF8.readString(UTF8.java:250)
>>>>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
>>>>>>>>>>>>>>>> 	at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:280)
>>>>>>>>>>>>>>>> 	at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75)
>>>>>>>>>>>>>>>> 	at org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
>>>>>>>>>>>>>>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
>>>>>>>>>>>>>>>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>>>>>>>>>>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>>>>>>> 	at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>>>> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1852)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1950)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1874)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>>>>>>>>>>>>>>>> 	at org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:135)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1795)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1754)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
>>>>>>>>>>>>>>>> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>>>>>>>>>>>>>>>> 	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39)
>>>>>>>>>>>>>>>> 	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61)
>>>>>>>>>>>>>>>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:153)
>>>>>>>>>>>>>>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>>>>>>>>>>>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>>>>>>>>>>>>> 	at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Dachuan Huang
>>>>>>>>>>>>>>> Cellphone: 614-390-7234
>>>>>>>>>>>>>>> 2015 Neil Avenue
>>>>>>>>>>>>>>> Ohio State University
>>>>>>>>>>>>>>> Columbus, Ohio
>>>>>>>>>>>>>>> U.S.A.
>>>>>>>>>>>>>>> 43210
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Dachuan Huang
>>>>>>>>>>>>> Cellphone: 614-390-7234
>>>>>>>>>>>>> 2015 Neil Avenue
>>>>>>>>>>>>> Ohio State University
>>>>>>>>>>>>> Columbus, Ohio
>>>>>>>>>>>>> U.S.A.
>>>>>>>>>>>>> 43210
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Dachuan Huang
>>>>>>>>>>> Cellphone: 614-390-7234
>>>>>>>>>>> 2015 Neil Avenue
>>>>>>>>>>> Ohio State University
>>>>>>>>>>> Columbus, Ohio
>>>>>>>>>>> U.S.A.
>>>>>>>>>>> 43210
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message