kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shantanu Deshmukh <shantanu...@gmail.com>
Subject Re: Effect of settings segment.ms and retention.ms not accurate
Date Wed, 30 May 2018 05:28:46 GMT
This is helpful. Thanks a lot :-)

On Tue, May 29, 2018 at 11:47 PM Matthias J. Sax <matthias@confluent.io>
wrote:

> ConsumerRecord#timestamp()
>
> similar to ConsumerRecord#key() and ConsumerRecord#value()
>
>
> -Matthias
>
> On 5/28/18 11:22 PM, Shantanu Deshmukh wrote:
> > But then I wonder, why such things are not mentioned anywhere in Kafka
> > configuration document? I relied on that setting and it caused us some
> > issues. If it is mentioned clearly then everyone will be aware. Could you
> > please point in right direction about reading timestamp of log message? I
> > will see about implementing that solution in code.
> >
> > On Tue, May 29, 2018 at 11:37 AM Matthias J. Sax <matthias@confluent.io>
> > wrote:
> >
> >> Retention time is a lower bound for how long it is guaranteed that data
> >> will be stored. This guarantee work "one way" only. There is no
> >> guarantee when data will be deleted after the bound passed.
> >>
> >> However, client side, you can always check the record timestamp and just
> >> drop older data that is still in the topic.
> >>
> >> Hope this helps.
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 5/28/18 9:18 PM, Shantanu Deshmukh wrote:
> >>> Please help.
> >>>
> >>> On Mon, May 28, 2018 at 5:18 PM Shantanu Deshmukh <
> shantanu88d@gmail.com
> >>>
> >>> wrote:
> >>>
> >>>>  I have a topic otp-sms. I want that retention of this topic should
> be 5
> >>>> minutes as OTPs are invalid post that amount of time. So I set
> >>>> retention.ms=300000. However, this was not working. So reading more
> in
> >>>> depth in Kafka configuration document I found another topic level
> >> setting
> >>>> that can be tuned for topic retention to work properly. So I set
> >>>> segment.ms=300000 as well.
> >>>>
> >>>> After these changes I saw that old logs go deleted. Still in the
> topic I
> >>>> could see one record which is more than 15 minutes old and not getting
> >>>> deleted. What does one have to do to actually delete messages
> generated
> >> n
> >>>> minutes ago?
> >>>>
> >>>
> >>
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message