spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sung Hwan Chung <coded...@cs.stanford.edu>
Subject Re: java.io.IOException Error in task deserialization
Date Fri, 10 Oct 2014 18:20:43 GMT
I can tell you what the environment and rough processes are like:

CDH5 Yarn
15 executors (16GB for driver, 8GB for executors)
Total cached data about 10GB
Shuffled data size per iteration ~1GB. - map followed by groupby followed
by map followed by collect
I'd imagine that every time map/groupby is called, the environment data
that get serialized to the mappers/groupbys are maxed at 250MB.
Periodic checkpointing



On Fri, Oct 10, 2014 at 10:34 AM, Davies Liu <davies@databricks.com> wrote:

> Maybe, TorrentBroadcast is more complicated than HttpBroadcast, could
> you tell us
> how to reproduce this issue? That will help us a lot to improve
> TorrentBroadcast.
>
> Thanks!
>
> On Fri, Oct 10, 2014 at 8:46 AM, Sung Hwan Chung
> <codedeft@cs.stanford.edu> wrote:
> > I haven't seen this at all since switching to HttpBroadcast. It seems
> > TorrentBroadcast might have some issues?
> >
> > On Thu, Oct 9, 2014 at 4:28 PM, Sung Hwan Chung <
> codedeft@cs.stanford.edu>
> > wrote:
> >>
> >> I don't think that I saw any other error message. This is all I saw.
> >>
> >> I'm currently experimenting to see if this can be alleviated by using
> >> HttpBroadcastFactory instead of TorrentBroadcast. So far, with
> >> HttpBroadcast, I haven't seen this recurring as of yet. I'll keep you
> >> posted.
> >>
> >> On Thu, Oct 9, 2014 at 4:21 PM, Davies Liu <davies@databricks.com>
> wrote:
> >>>
> >>> This exception should be caused by another one, could you paste all of
> >>> them here?
> >>>
> >>> Also, that will be great if you can provide a script to reproduce this
> >>> problem.
> >>>
> >>> Thanks!
> >>>
> >>> On Fri, Sep 26, 2014 at 6:11 AM, Arun Ahuja <aahuja11@gmail.com>
> wrote:
> >>> > Has anyone else seen this erorr in task deserialization?  The task
is
> >>> > processing a small amount of data and doesn't seem to have much data
> >>> > hanging
> >>> > to the closure?  I've only seen this with Spark 1.1
> >>> >
> >>> > Job aborted due to stage failure: Task 975 in stage 8.0 failed 4
> times,
> >>> > most
> >>> > recent failure: Lost task 975.3 in stage 8.0 (TID 24777, host.com):
> >>> > java.io.IOException: unexpected exception type
> >>> >
> >>> >
> >>> >
> java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538)
> >>> >
> >>> >
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1025)
> >>> >
> >>> > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> >>> >
> >>> >
> >>> >
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> >>> >
> >>> > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> >>> >
> >>> >
> >>> >
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> >>> >
> >>> > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> >>> >
> >>> >
> >>> >
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> >>> >
> >>> > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> >>> >
> >>> > java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> >>> >
> >>> >
> >>> >
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> >>> >
> >>> >
> >>> >
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> >>> >
> >>> > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
> >>> >
> >>> >
> >>> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>> >
> >>> >
> >>> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>> >         java.lang.Thread.run(Thread.java:744)
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >>> For additional commands, e-mail: user-help@spark.apache.org
> >>>
> >>
> >
>

Mime
View raw message