spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <ilike...@gmail.com>
Subject Re: Akka disassociation on Java SE Embedded
Date Sun, 01 Jun 2014 17:06:48 GMT
Thanks for the update! I've also run into the block manager timeout issue,
it might be a good idea to increase the default significantly (it would
probably timeout immediately if the TCP connection itself dropped anyway).


On Sun, Jun 1, 2014 at 9:48 AM, Chanwit Kaewkasi <chanwit@gmail.com> wrote:

> Hi all,
>
> This is what I found:
>
> 1. Like Aaron suggested, an executor will be killed silently when the
> OS's memory is running out.
> I've found this many times to conclude this it's real. Adding swap and
> increasing the JVM heap solved the problem, but you will encounter OS
> paging out and full GC.
>
> 2. OS paging out and full GC are not likely to affect my benchmark
> much while processing data from HDFS. But Akka process's randomly
> killed during the network-related stage (for example, sorting). I've
> found that an Akka process cannot fetch the result fast enough.
> Increasing the block manager timeout helped a lot. I've doubled the
> value many times as the network of our ARM cluster is quite slow.
>
> 3. We'd like to collect times spent for all stages of our benchmark.
> So we always re-run when some tasks failed. Failure happened a lot but
> it's understandable as Spark is designed on top of Akka's let-it-crash
> philosophy. To make the benchmark run more perfectly (without a task
> failure), I called .cache() before calling the transformation of the
> next stage. And it helped a lot.
>
> Combined above and others tuning, we can now boost the performance of
> our ARM cluster to 2.8 times faster than our first report.
>
> Best regards,
>
> -chanwit
>
> --
> Chanwit Kaewkasi
> linkedin.com/in/chanwit
>
>
> On Wed, May 28, 2014 at 1:13 AM, Chanwit Kaewkasi <chanwit@gmail.com>
> wrote:
> > May be that's explaining mine too.
> > Thank you very much, Aaron !!
> >
> > Best regards,
> >
> > -chanwit
> >
> > --
> > Chanwit Kaewkasi
> > linkedin.com/in/chanwit
> >
> >
> > On Wed, May 28, 2014 at 12:47 AM, Aaron Davidson <ilikerps@gmail.com>
> wrote:
> >> Spark should effectively turn Akka's failure detector off, because we
> >> historically had problems with GCs and other issues causing
> disassociations.
> >> The only thing that should cause these messages nowadays is if the TCP
> >> connection (which Akka sustains between Actor Systems on different
> machines)
> >> actually drops. TCP connections are pretty resilient, so one common
> cause of
> >> this is actual Executor failure -- recently, I have experienced a
> >> similar-sounding problem due to my machine's OOM killer terminating my
> >> Executors, such that they didn't produce any error output.
> >>
> >>
> >> On Thu, May 22, 2014 at 9:19 AM, Chanwit Kaewkasi <chanwit@gmail.com>
> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> On an ARM cluster, I have been testing a wordcount program with JRE 7
> >>> and everything is OK. But when changing to the embedded version of
> >>> Java SE (Oracle's eJRE), the same program cannot complete all
> >>> computing stages.
> >>>
> >>> It is failed by many Akka's disassociation.
> >>>
> >>> - I've been trying to increase Akka's timeout but still stuck. I am
> >>> not sure what is the right way to do so? (I suspected that GC pausing
> >>> the world is causing this).
> >>>
> >>> - Another question is that how could I properly turn on Akka's logging
> >>> to see what's the root cause of this disassociation problem? (If my
> >>> guess about GC is wrong).
> >>>
> >>> Best regards,
> >>>
> >>> -chanwit
> >>>
> >>> --
> >>> Chanwit Kaewkasi
> >>> linkedin.com/in/chanwit
> >>
> >>
>

Mime
View raw message