kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tejas Patil <tejas.patil...@gmail.com>
Subject Re: reprocessing messages in kafka
Date Thu, 01 Aug 2013 04:54:14 GMT
As @Milind said, it is possible that a consumer consumes the same message
more than once.
This happens when there is an unclean shutdown of the consumer and it is
not able to commit its latest offset to Zookeeper. When the failed consumer
comes up, it would fetch the stale offset from zookeeper thus re-consuming
a small set of messages.


On Wed, Jul 31, 2013 at 3:31 PM, Milind Parikh <milindparikh@gmail.com>wrote:

> It is possible to consume the same message more than once with the same
> consumer. However WHAT you actually do with the message (such as idempotent
> writes) is the tricker part.
>
> Regards
> Milind
>
>
>
> On Wed, Jul 31, 2013 at 8:22 AM, Oleg Ruchovets <oruchovets@gmail.com
> >wrote:
>
> > Hi ,
> >
> > I just don't know which mail list is correct to post this question( storm
> > or kafka)? Sorry for cross post.
> >
> >     I just read the documentation which describe guaranteed message
> > processing with storm -
> > https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing
> .
> >
> > The question actually what will be with the message which was consumed by
> > storm and it is failed to process. In case I'll use anchoring technique ,
> > trying to process the message the second time:  will this be available in
> > kafka ( I am using storm-kafka spout)?
> >
> > I mean Is it possible to consume the same message in kafka more then one
> > time with the same consumer?
> >
> > Thanks
> > Oleg.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message