kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Akhtar <ali.rac...@gmail.com>
Subject Re: Deleting a message after all consumer have consumed it
Date Sat, 08 Oct 2016 01:18:42 GMT
Also, you can set a retention period and have messages get auto deleted
after a certain time (default 1 week)

On Sat, Oct 8, 2016 at 3:21 AM, Hans Jespersen <hans@confluent.io> wrote:

> Kafka doesn’t work that way. Kafka is “Publish-subscribe messaging
> rethought as a distributed commit log”. The messages in the log do not get
> deleted just because “all" clients have consumed the messages. Besides you
> could always have a late joining consumer come along and if you mistakenly
> deleted the messages then they wouldn’t be there for any new consumers.
> Ditto for replaying messages back to consumers after they consume the
> messages the first time. Kafka just keeps the messages and once consumed
> each client moves their offset forward to point to the next message. You
> can tell is any consumer has finished processing a message by checking the
> offsets for that client. For the older clients these are stored in
> Zookeeper but for the new consumers (0.9+) they are in a special kafka
> topic dedicated to storing client offsets.
>
>
> -hans
>
>
>
> > On Oct 7, 2016, at 1:34 PM, Hysi, Lorenc <lorenc.hysi@boeing.com> wrote:
> >
> > Hello,
> >
> > Thank you for your time.
> >
> > I wanted to ask whether it's possible to remove a message from a topic
> after making sure all consumers have gotten a it. If so what is the best
> way to achieve this?
> >
> > Also how do I make sure that all consumers  have received a message. Any
> way to do this in kafka or zookeeper? I keep looking but I'm only seeing
> that only consumers themselves keep track of where they are and if so any
> way to have consumers talk to each other to make sure they all have
> consumed the same message.
> >
> > If this is not the right place to ask this question I apologize in
> advance and please let me know where to ask if you can.
> >
> > Thank you!
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message