spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jonathan.keebler" <jkeeble...@gmail.com>
Subject Re: Spark stalling during shuffle (maybe a memory issue)
Date Tue, 20 May 2014 04:23:45 GMT
Thanks for the response, Aaron!  We'll give it a try tomorrow.


On Tue, May 20, 2014 at 12:13 AM, Aaron Davidson [via Apache Spark User
List] <ml-node+s1001560n6073h83@n3.nabble.com> wrote:

> This is very likely because the serialized map output locations buffer
> exceeds the akka frame size. Please try setting "spark.akka.frameSize"
> (default 10 MB) to some higher number, like 64 or 128.
>
> In the newest version of Spark, this would throw a better error, for what
> it's worth.
>
>
>
> On Mon, May 19, 2014 at 8:39 PM, jonathan.keebler <[hidden email]<http://user/SendEmail.jtp?type=node&node=6073&i=0>
> > wrote:
>
>> Has anyone observed Spark worker threads stalling during a shuffle phase
>> with
>> the following message (one per worker host) being echoed to the terminal
>> on
>> the driver thread?
>>
>> INFO spark.MapOutputTrackerActor: Asked to send map output locations for
>> shuffle 0 to [worker host]...
>>
>>
>> At this point Spark-related activity on the hadoop cluster completely
>> halts
>> .. there's no network activity, disk IO or CPU activity, and individual
>> tasks are not completing and the job just sits in this state.  At this
>> point
>> we just kill the job & a re-start of the Spark server service is required.
>>
>> Using identical jobs we were able to by-pass this halt point by increasing
>> available heap memory to the workers, but it's odd we don't get an
>> out-of-memory error or any error at all.  Upping the memory available
>> isn't
>> a very satisfying answer to what may be going on :)
>>
>> We're running Spark 0.9.0 on CDH5.0 in stand-alone mode.
>>
>> Thanks for any help or ideas you may have!
>>
>> Cheers,
>> Jonathan
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-stalling-during-shuffle-maybe-a-memory-issue-tp6067.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-stalling-during-shuffle-maybe-a-memory-issue-tp6067p6073.html
>  To unsubscribe from Spark stalling during shuffle (maybe a memory issue), click
> here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=6067&code=amtlZWJsZXI0MkBnbWFpbC5jb218NjA2N3wtMjA5NzAzMzE5NQ==>
> .
> NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-stalling-during-shuffle-maybe-a-memory-issue-tp6067p6074.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message