kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rosenberg <...@squareup.com>
Subject Re: question about new consumer offset management in 0.8.2
Date Thu, 05 Feb 2015 20:40:07 GMT
Ok, so it looks like the default settings are:

offset.storage = zookeeper
dual.commit.enabled = true

The doc for 'dual.commit.enabled' seems to imply (but doesn't clearly
state) that it will only apply if offset.storage = kafka.  Is that right?
(I'm guessing not)

"*If you are using "kafka"* as offsets.storage, you can dual commit offsets
to ZooKeeper (in addition to Kafka)."

It seems to me less than ideal to have the default behavior to have
dual.commit.enabled = true, since this seems like a performance hit, no?
I'd think you'd only want this during a planned migration.

Also, I assume it's desirable to switch to using 'kafka' for offset
storage, for performance reasons?  Will it better handle a larger number of
topics?
Also, I assume the __consumer_offsets topic will be set to have an infinite
retention policy internally, is that right?  So that currently committed
offsets for a given consumer group won't be lost?

Jason

On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:

> This is documented in the official docs:
> http://kafka.apache.org/documentation.html#distributionimpl
>
> On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote:
> > What are the defaults for those settings (I assume it will be to continue
> > using only zookeeper by default)?
> >
> > Also, if I have a cluster of consumers sharing the same groupId, and I
> > update them via a rolling release, will it be a problem during the
> rolling
> > restart if there is inconsistency in the settings for a short time?  Or
> is
> > it required that the entire cluster be stopped, then update configs, then
> > restart all nodes?
> >
> > Jason
> >
> > On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gshapira@cloudera.com>
> wrote:
> >
> > > Thanks Jon. I updated the FAQ with your procedure:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2
> > > ?
> > >
> > > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst <
> > > jbringhurst@linkedin.com.invalid> wrote:
> > >
> > > > There should probably be a wiki page started for this so we have the
> > > > details in one place. The same question was asked on Freenode IRC a
> few
> > > > minutes ago. :)
> > > >
> > > > A summary of the migration procedure is:
> > > >
> > > > 1) Upgrade your brokers and set dual.commit.enabled=false and
> > > > offsets.storage=zookeeper (Commit offsets to Zookeeper Only).
> > > > 2) Set dual.commit.enabled=true and offsets.storage=kafka and restart
> > > > (Commit offsets to Zookeeper and Kafka).
> > > > 3) Set dual.commit.enabled=false and offsets.storage=kafka and
> restart
> > > > (Commit offsets to Kafka only).
> > > >
> > > > -Jon
> > > >
> > > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jbr@squareup.com>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > For 0.8.2, one of the features listed is:
> > > > >  - Kafka-based offset storage.
> > > > >
> > > > > Is there documentation on this (I've heard discussion of it of
> course)?
> > > > >
> > > > > Also, is it something that will be used by existing consumers when
> they
> > > > > migrate up to 0.8.2?  What is the migration process?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jason
> > > >
> > > >
> > >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message