spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Albert <m_albert...@yahoo.com.INVALID>
Subject Re: How to avoid executor time out on yarn spark while dealing with large shuffle skewed data?
Date Fri, 21 Aug 2015 10:54:29 GMT
This is something of a wild guess, but I find that when executors start disappearingfor no
obvious reason, this is usually because the yarn node-managers have decided that the containers
are using too much memory and then terminate the executors.
Unfortunately, to see evidence of this, one needs to carefully review the yarn node-manager
logson the workers -- it doesn't seem to show up in the UI.
What I generally do is some combination of:   1) increasing executors memory (often also
decreasing number of executors)   2) decreasing the number of cores per executor   3)
increase the executor memory overhead.
Good luck.
-Mike
      From: Sandy Ryza <sandy.ryza@cloudera.com>
 To: Umesh Kacha <umesh.kacha@gmail.com> 
Cc: "user@spark.apache.org" <user@spark.apache.org> 
 Sent: Thursday, August 20, 2015 5:21 PM
 Subject: Re: How to avoid executor time out on yarn spark while dealing with large shuffle
skewed data?
   
GC wouldn't necessarily result in errors - it could just be slowing down your job and causing
the executor JVMs to stall.  If you click on a stage in the UI, you should end up on a page
with all the metrics concerning the tasks that ran in that stage.  "GC Time" is one of these
task metrics.
-Sandy


On Thu, Aug 20, 2015 at 8:54 AM, Umesh Kacha <umesh.kacha@gmail.com> wrote:

Hi where do I see GC time in UI? I have set spark.yarn.executor.memoryOverhead as 3500 which
seems to be good enough I believe. So you mean only GC could be the reason behind timeout
I checked Yarn logs I did not see any GC error there. Please guide. Thanks much.
On Thu, Aug 20, 2015 at 8:14 PM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:

Moving this back onto user@
Regarding GC, can you look in the web UI and see whether the "GC time" metric dominates the
amount of time spent on each task (or at least the tasks that aren't completing)?
Also, have you tried bumping your spark.yarn.executor.memoryOverhead?  YARN may be killing
your executors for using too much off-heap space.  You can see whether this is happening
by looking in the Spark AM or YARN NodeManager logs.
-Sandy
On Thu, Aug 20, 2015 at 7:39 AM, Umesh Kacha <umesh.kacha@gmail.com> wrote:

Hi thanks much for the response. Yes I tried default settings too 0.2 it was also going into
timeout if it is spending time in GC then why it is not throwing GC error I don't see any
such error. Yarn logs are not helpful at all. What is tungsten how do I use it? Spark is doing
great I believe my job runs successfully and 60% tasks completes only after first executor
gets lost things are messing. On Aug 20, 2015 7:59 PM, "Sandy Ryza" <sandy.ryza@cloudera.com>
wrote:

What sounds most likely is that you're hitting heavy garbage collection.  Did you hit issues
when the shuffle memory fraction was at its default of 0.2?  A potential danger with setting
the shuffle storage to 0.7 is that it allows shuffle objects to get into the GC old generation,
which triggers more stop-the-world garbage collections.
Have you tried enabling Tungsten / unsafe?
Unfortunately, Spark is still not that great at dealing with heavily-skewed shuffle data,
because its reduce-side aggregation still operates on Java objects instead of binary data.
-Sandy
On Thu, Aug 20, 2015 at 7:21 AM, Umesh Kacha <umesh.kacha@gmail.com> wrote:

Hi Sandy thanks much for the response. I am using Spark 1.4.1 and I have set spark.shuffle.storage
as 0.7 as my spark job involves 4 groupby queries executed using hiveContext.sql my data set
is skewed so will be more shuffling I believe I don't know what's wrong spark job runs fine
for almost an hour and when shuffle read shuffle write column in UI starts to show more than
10 gb executor starts to getting lost because of timeout and slowly other executor starts
getting lost. Please guide. On Aug 20, 2015 7:38 PM, "Sandy Ryza" <sandy.ryza@cloudera.com>
wrote:

What version of Spark are you using?  Have you set any shuffle configs?
On Wed, Aug 19, 2015 at 11:46 AM, unk1102 <umesh.kacha@gmail.com> wrote:

I have one Spark job which seems to run fine but after one hour or so
executor start getting lost because of time out something like the following
error

cluster.yarnScheduler : Removing an executor 14 650000 timeout exceeds
600000 seconds

and because of above error couple of chained errors starts to come like
FetchFailedException, Rpc client disassociated, Connection reset by peer,
IOException etc

Please see the following UI page I have noticed when shuffle read/write
starts to increase more than 10 GB executors starts getting lost because of
timeout. How do I clear this stacked memory of 10 GB in shuffle read/write
section I dont cache anything why Spark is not clearing those memory. Please
guide.

IMG_20150819_231418358.jpg
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n24345/IMG_20150819_231418358.jpg>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-avoid-executor-time-out-on-yarn-spark-while-dealing-with-large-shuffle-skewed-data-tp24345.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org















  
Mime
View raw message