spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Walrus theCat <walrusthe...@gmail.com>
Subject Re: cluster hangs for no apparent reason
Date Sat, 02 Nov 2013 02:15:07 GMT
That's what I thought, too.  So is it not "hanging", just recalculating for
a very long time?  The log stops updating and it just gives the output I
posted.  If there are any suggestions as to parameters to change, or any
other data, it would be appreciated.

Thank you, Shangyu.


On Fri, Nov 1, 2013 at 11:31 AM, Shangyu Luo <lsyurd@gmail.com> wrote:

> I think the missing parent may be not abnormal. From my understanding,
> when a Spark task cannot find its parent, it can use some meta data to find
> the result of its parent or recalculate its parent's value. Imaging in a
> loop, a Spark task tries to find some value from the last iteration's
> result.
>
>
> 2013/11/1 Walrus theCat <walrusthecat@gmail.com>
>
>> Are there heuristics to check when the scheduler says it is "missing
>> parents" and just hangs?
>>
>>
>>
>> On Thu, Oct 31, 2013 at 4:56 PM, Walrus theCat <walrusthecat@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I'm not sure what's going on here.  My code seems to be working thus far
>>> (map at SparkLR:90 completed.)  What can I do to help the scheduler out
>>> here?
>>>
>>> Thanks
>>>
>>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: Completed
>>> ShuffleMapTask(10, 211)
>>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: Stage 10 (map at
>>> SparkLR.scala:90) finished in 0.923 s
>>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: looking for newly
>>> runnable stages
>>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: running: Set(Stage 11)
>>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: waiting: Set(Stage 9,
>>> Stage 8)
>>> 13/10/31 02:10:13 INFO scheduler.DAGScheduler: failed: Set()
>>> 13/10/31 02:10:16 INFO scheduler.DAGScheduler: Missing parents for Stage
>>> 9: List(Stage 11)
>>> 13/10/31 02:10:16 INFO scheduler.DAGScheduler: Missing parents for Stage
>>> 8: List(Stage 9)
>>>
>>>
>>>
>>>
>>
>
>
> --
> --
>
> Shangyu, Luo
> Department of Computer Science
> Rice University
>
> --
> Not Just Think About It, But Do It!
> --
> Success is never final.
> --
> Losers always whine about their best
>

Mime
View raw message