kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Penner <j...@longtailvideo.com>
Subject Re: JAVA HEAP settings for KAFKA in production
Date Wed, 07 May 2014 17:31:18 GMT
Just curious if these JAVA_HEAP tunings are still for machines listed in 6.4
Hardware and OS <http://kafka.apache.org/documentation.html#hwandos>:
"We are using dual quad-core Intel Xeon machines with 24GB of memory."

Seems like a small heap size for 24GB of memory, or is that done to
maximize as much memory as possible for pagecache?

-penner


On Tue, May 6, 2014 at 1:50 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Updated the docs with Todd's recommendation -
> http://kafka.apache.org/081/documentation.html#java
>
> Thanks,
> Neha
>
>
> On Tue, May 6, 2014 at 9:40 AM, Jay Kreps <jay.kreps@gmail.com> wrote:
>
> > Hey Todd,
> >
> > Doc patch? :-)
> >
> > svn co http://svn.apache.org/repos/asf/kafka/site/081/
> >
> > Don't stress about html or formatting, I'm happy to do that part. I would
> > love to give people more authoritative advice. Right now everything is a
> > bit obsolete and wrong.
> >
> > -Jay
> >
> >
> > On Mon, May 5, 2014 at 10:36 PM, Todd Palino <tpalino@linkedin.com>
> wrote:
> >
> > > I apologize for taking a couple days to jump in on this. We¹re
> currently
> > > running JDK 1.7 u51, and we¹ve switched over to the G1 collector. If
> you
> > > do this (and I highly recommend it), make sure you¹re on u51. We tried
> > out
> > > u21 in testing, but we had a number of problems with the GC
> > implementation
> > > in that version.
> > >
> > > Our tuning looks like this:
> > > -Xms4g -Xmx4g -XX:PermSize=48m -XX:MaxPermSize=48m -XX:+UseG1GC
> > > -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
> > >
> > > For reference, I¹m looking at the stats on one of our busiest clusters
> > (at
> > > peak):
> > >         - 15 brokers
> > >         - 15.5k partitions (replication factor 2)
> > >
> > >         - 400k messages/sec in
> > >         - 70 MB/sec inbound, 400 MB/sec+ outbound
> > >
> > > The tuning looks fairly aggressive, but all of our brokers in that
> > cluster
> > > have a 90% GC pause time of about 21ms, and they¹re doing less than 1
> > > young GC per second. We haven¹t seen a single full GC on those brokers
> in
> > > the last month, and previous to that I think we only saw them when I
> was
> > > messing around with the cluster in a very painful way, not under
> anything
> > > approaching normal traffic.
> > >
> > > -Todd
> > >
> > >
> > > On 5/1/14, 9:21 PM, "Neha Narkhede" <neha.narkhede@gmail.com> wrote:
> > >
> > > >The GC settings at http://kafka.apache.org/documentation.html#javaare
> > > >old.
> > > >We meant to update the documentation with the new GC settings using
> the
> > G7
> > > >collector, but we haven't gotten around to doing that. Let me reach
> out
> > to
> > > >our engineer, Todd Palino, who worked on tuning GC for Kafka at
> LinkedIn
> > > >to
> > > >see if we can update our docs.
> > > >
> > > >Thanks,
> > > >Neha
> > > >
> > > >
> > > >On Thu, May 1, 2014 at 9:02 PM, Jun Rao <junrao@gmail.com> wrote:
> > > >
> > > >> http://kafka.apache.org/documentation.html#java
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >>
> > > >> On Thu, May 1, 2014 at 12:19 PM, Cassa L <lcassa8@gmail.com>
wrote:
> > > >>
> > > >> > Hi,
> > > >> >   I want to know what usually, are the JAVA_HEAP settings
> > recommended
> > > >>for
> > > >> > kafka servers in production.
> > > >> >
> > > >> > Thanks
> > > >> > LCassa
> > > >> >
> > > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message