kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Luban <michael.lu...@gmail.com>
Subject Re: Specifying initial message offset with ZookeeperConnectorConsumer
Date Sat, 18 Feb 2012 02:54:54 GMT
autooffset.reset="largest" in conjunction with deleting the corresponding
zookeeper offsets solved the problem.  Thanks to both for sharing this
excellent advice.

On Mon, Jan 23, 2012 at 8:12 PM, Joel Koshy <jjkoshy@gmail.com> wrote:

> Largest works - however, autooffset only applies if the consumer group
> is new (non-existent in ZK) or if the stored offset is out of range.
> If you restart your consumer at a later point it will resume from the
> last check-pointed offset. So from the scenario you describe (down for
> a period of time...) it appears that it wouldn't be enough for you.
> Depending on the details of your set up, one thing you could do is to
> clean up the zookeeper entries for your consumer group before bringing
> up your consumer service again.
>
> Joel
>
> On Mon, Jan 23, 2012 at 4:49 PM, Evan Chan <ev@ooyala.com> wrote:
> > Michael,
> >
> > It seems there is a property "autooffset.reset" here:
> > http://incubator.apache.org/kafka/configuration.html
> >
> > which you can set to "largest".   I haven't used it though so I can't be
> > sure.
> >
> > -Evan
> >
> >
> > On Mon, Jan 23, 2012 at 4:12 PM, Michael Luban <michael.luban@gmail.com
> >wrote:
> >
> >> I'm interested in having my consumer service start consuming the most
> >> recent message rather than the message corresponding to the last offset
> >> stored.  For instance, if the kafka consumer service is down for a
> period
> >> of time, I would like to skip over all messages since the last recorded
> >> offset and begin consuming from the most recent message.
> >>
> >> According to the following, it seems I have to use SimpleConsumer to
> >> achieve this?  Seems like this would be difficult to manage in a "load
> >> balanced" consumer scenario (federation of consumers in the same group
> >> consuming the same topic).
> >>
> >>
> https://groups.google.com/group/kafka-dev/browse_thread/thread/ee7fe5f8eef49a4d/a4469233880a7f76?lnk=gst&q=offset#a4469233880a7f76
> >>
> >> Any additional insight appreciated.
> >>
> >
> >
> >
> > --
> > --
> > *Evan Chan*
> > Senior Software Engineer |
> > ev@ooyala.com | (650) 996-4600
> > www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > @ooyala<http://www.twitter.com/ooyala>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message