spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Binh Nguyen <ngb...@gmail.com>
Subject Re: GC tuning for Spark
Date Thu, 16 Jan 2014 20:34:02 GMT
I think incorporating https://github.com/amplab/tachyon/wiki is a better
solution. I remembered Matei has said that it was in his plan but not sure
about the ETA for it to happen.


On Thu, Jan 16, 2014 at 12:30 PM, Mark Hamstra <mark@clearstorydata.com>wrote:

> And, of course, there are the bigger-hammer-than-GC-tuning approaches using
> some combination of unchecked, off-heap and Tachyon.
>
>
> On Thu, Jan 16, 2014 at 11:54 AM, Tathagata Das <
> tathagata.das1565@gmail.com
> > wrote:
>
> > There are a bunch of tricks noted in the Tuning
> > Guide<
> > http://spark.incubator.apache.org/docs/latest/tuning.html#memory-tuning
> >.
> > You may have seen them already but I thought its still worth mentioning
> for
> > the records.
> >
> > Besides those, if you are concerned about consistent latency (that is,
> low
> > variability in the job processing times), then using
> > concurrent-mark-and-sweep GC is recommended. Instead of big
> stop-the-world
> > GC pauses, there are many smaller pauses. This reduction in variability
> > comes at the cost of processing throughput though, so thats a tradeoff.
> >
> > TD
> >
> >
> > On Thu, Jan 16, 2014 at 11:35 AM, Kay Ousterhout <keo@eecs.berkeley.edu
> > >wrote:
> >
> > > Hi all,
> > >
> > > I'm finding that Java GC can be a major performance bottleneck when
> > running
> > > Spark at high (>50% or so) memory utilization.  What GC tuning have
> > people
> > > tried for Spark and how effective has it been?
> > >
> > > Thanks!
> > >
> > > Kay
> > >
> >
>



-- 

Binh Nguyen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message