kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: question about new consumer offset management in 0.8.2
Date Fri, 06 Feb 2015 18:28:46 GMT

On Thu, Feb 05, 2015 at 11:57:15PM -0800, Joel Koshy wrote:
> On Fri, Feb 06, 2015 at 12:43:37AM -0500, Jason Rosenberg wrote:
> > I'm not sure what you mean by 'default' behavior 'only if' offset.storage
> > is kafka.  Does that mean the 'default' behavior is 'false' if
> > offset.storage is 'zookeeper'?  Can that be clarified in the config
> > documentation section?
> > 
> > In section 5.6 where the offset managements is described, there is this:
> > "A roll-back (i.e., migrating from Kafka back to ZooKeeper) can also be
> > performed using the above steps if you set offsets.storage=zookeeper."
> > 
> > This implies that dual commit will work also if offsets.storage=zookeeper,
> > no?  Just not by default?  Perhaps there needs to be clarification there
> > (and in the config section for offsets.storage & dual.commit.enabled).
> Actually I think there may be a bug here if someone needs to roll back
> from Kafka-based offsets to zookeeper. Will reply tomorrow on this.
> > 

Never mind - I think we are fine here.  The scenario I was thinking
about is the following: 
- If there are three consumer instances c0, c1, c2 consuming
  partitions pX, pY, ...  and are committing offsets to Kafka and you
  want to migrate to zookeeper
- Do a rolling bounce to turn on dual-commit (and keep offset.storage
  = kafka)
- Do another rolling bounce to set offset.storage to zookeeper:
  - Say, you bounce c0 to commit offsets to zk and it comes back up
    and then owns pX. It begins to commit offsets for pX to zookeeper
  - You then bounce c1; after it goes down due to our partition
    assignment strategy say pX is now assigned to c2 (which has not
    yet been bounced).
  - c2 uses offset.storage=kafka so would fetch a potentially stale
    offset for pX which would be an issue. 
  - So we explicitly handle this case - if dual.commit is turned on
    and offset.storage is kafka, then the broker fetches offsets from
    both Kafka and ZooKeeper and selects the maximum of the two.

Let me know if you see any holes in the above.

dual.commit is confusing and would have been (slightly) less confusing if it
was called offset.migration.in.progress or something similar. Still, I think
we can document the process carefully and state clearly that it is
intended for use during migration/roll-back only.

View raw message