spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayur Rustagi <mayur.rust...@gmail.com>
Subject Re: Spark worker threads waiting
Date Wed, 19 Mar 2014 16:25:42 GMT
You could have some outlier task that is preventing the next set of stages
from launching. Can you check out stages state in the Spark WebUI, is any
task running or is everything halted.
Regards
Mayur

Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Wed, Mar 19, 2014 at 5:40 AM, Domen Grabec <domen@celtra.com> wrote:

> Hi,
>
> I have a cluster with 16 nodes, each node has 69Gb ram (50GB goes to
> spark) and 8 cores running spark 0.8.1. I have a groupByKey operation that
> causes a wide RDD dependency so shuffle write and shuffle read are
> performed.
>
> For some reason all worker threads seem to sleep for about 3-4 minutes
> each time performing a shuffle read and completing a set of tasks. See
> graphs below how no resources are being utilized in specific time windows.
>
> Each time 3-4 minutes pass, a next set of tasks are being grabbed and
> processed, and then another waiting period happens.
>
> Each task has an input of 80Mb +- 5Mb data to shuffle read.
>
>  [image: Inline image 1]
>
> Here <http://pastebin.com/UHWMdTRY> is a link to thread dump performed in
> the middle of the waiting period. Any idea what could cause the long waits?
>
> Kind regards, Domen
>

Mime
View raw message