kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: question about new consumer offset management in 0.8.2
Date Fri, 06 Feb 2015 07:57:15 GMT
On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote:
> I'm not sure what you mean by 'default' behavior 'only if' offset.storage
> is kafka.  Does that mean the 'default' behavior is 'false' if
> offset.storage is 'zookeeper'?  Can that be clarified in the config
> documentation section?
> 
> In section 5.6 where the offset managements is described, there is this:
> "A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be
> performed using the above steps if you set offsets.storage=zookeeper."
> 
> This implies that dual commit will work also if offsets.storage=zookeeper,
> no?  Just not by default?  Perhaps there needs to be clarification there
> (and in the config section for offsets.storage & dual.commit.enabled).

Actually I think there may be a bug here if someone needs to roll back
from Kafka-based offsets to zookeeper. Will reply tomorrow on this.

> 
> The doc in section 5.6 is probably in need of editing, it looks like it in
> places assumes zookeeper offset storage, and has some repeated sentences,
> etc.

Will do.

> 
> Finally, why is section 5.6 titled "Distribution"?  Seems to be a grab-bag
> of mostly consumer related topics?

Yes this was prior structure that can be improved.

> >
> > > On Thu, Feb 5, 2015 at 2:21 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:
> > >
> > > > This is documented in the official docs:
> > > > http://kafka.apache.org/documentation.html#distributionimpl
> > > >
> > > > On Thu, Feb 05, 2015 at 01:23:01PM -0500, Jason Rosenberg wrote:
> > > > > What are the defaults for those settings (I assume it will be to
> > continue
> > > > > using only zookeeper by default)?
> > > > >
> > > > > Also, if I have a cluster of consumers sharing the same groupId,
and
> > I
> > > > > update them via a rolling release, will it be a problem during the
> > > > rolling
> > > > > restart if there is inconsistency in the settings for a short time?
> > Or
> > > > is
> > > > > it required that the entire cluster be stopped, then update configs,
> > then
> > > > > restart all nodes?
> > > > >
> > > > > Jason
> > > > >
> > > > > On Thu, Feb 5, 2015 at 12:45 PM, Gwen Shapira <gshapira@cloudera.com
> > >
> > > > wrote:
> > > > >
> > > > > > Thanks Jon. I updated the FAQ with your procedure:
> > > > > >
> > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdowemigratetocommittingoffsetstoKafka(ratherthanZookeeper)in0.8.2
> > > > > > ?
> > > > > >
> > > > > > On Thu, Feb 5, 2015 at 9:16 AM, Jon Bringhurst <
> > > > > > jbringhurst@linkedin.com.invalid> wrote:
> > > > > >
> > > > > > > There should probably be a wiki page started for this so
we have
> > the
> > > > > > > details in one place. The same question was asked on Freenode
> > IRC a
> > > > few
> > > > > > > minutes ago. :)
> > > > > > >
> > > > > > > A summary of the migration procedure is:
> > > > > > >
> > > > > > > 1) Upgrade your brokers and set dual.commit.enabled=false
and
> > > > > > > offsets.storage=zookeeper (Commit offsets to Zookeeper
Only).
> > > > > > > 2) Set dual.commit.enabled=true and offsets.storage=kafka
and
> > restart
> > > > > > > (Commit offsets to Zookeeper and Kafka).
> > > > > > > 3) Set dual.commit.enabled=false and offsets.storage=kafka
and
> > > > restart
> > > > > > > (Commit offsets to Kafka only).
> > > > > > >
> > > > > > > -Jon
> > > > > > >
> > > > > > > On Feb 5, 2015, at 9:03 AM, Jason Rosenberg <jbr@squareup.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > For 0.8.2, one of the features listed is:
> > > > > > > >  - Kafka-based offset storage.
> > > > > > > >
> > > > > > > > Is there documentation on this (I've heard discussion
of it of
> > > > course)?
> > > > > > > >
> > > > > > > > Also, is it something that will be used by existing
consumers
> > when
> > > > they
> > > > > > > > migrate up to 0.8.2?  What is the migration process?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jason
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> >
> > --
> > Joel
> >


Mime
View raw message