kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhimanyu Nagrath <abhimanyunagr...@gmail.com>
Subject Re: Messages are repeating in kafka
Date Wed, 24 May 2017 06:40:29 GMT
Hi Kant,

After setting this configuration offsets.retention.minutes . I am in doubt
about the two things
 1. If I am deleting a topic will that topic offsets would also get deleted
or will they present for 30 days?
 2. What will happen if for some topics my log.retention.hours = 168 and
offsets.retention.minutes= 1440 * 30 ?

Regards,
Abhimanyu

On Mon, May 22, 2017 at 4:09 PM, Abhimanyu Nagrath <
abhimanyunagrath@gmail.com> wrote:

> @Kant I was going through the offset related configurations before setting
> offsets.retention.minutes so came accross this configuration and thought to
> ask whether this should also be tuned or not.
>
>
> Regards,
> Abhimanyu
>
>
>
>
> On Mon, May 22, 2017 at 2:24 PM, kant kodali <kanth909@gmail.com> wrote:
>
>> @Abhimanyu Why do you think you need to set that? Did you try setting
>> offsets.retention.minutes
>> = 1440 * 30 and still seeing duplicates?
>>
>> On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath <
>> abhimanyunagrath@gmail.com> wrote:
>>
>> > Hi Girish ,
>> >
>> > Do I need to tune this configuration offsets.retention.check.interv
>> al.ms
>> > also . Please let me know if I need to tune any other configuration.
>> >
>> >
>> > Regards,
>> > Abhimanyu
>> >
>> > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <girishaher@gmail.com>
>> wrote:
>> >
>> > > Yup, exactly as Kant said.
>> > > Also make sure that the retention of the offsets topic is an upper
>> bound
>> > > across all topics. So in this case, don't create any other topics in
>> the
>> > > future with retention of more than 30 days or otherwise they may have
>> the
>> > > same problem too.
>> > >
>> > > On May 21, 2017 03:25, "Abhimanyu Nagrath" <
>> abhimanyunagrath@gmail.com>
>> > > wrote:
>> > >
>> > >> Hi Kant,
>> > >>
>> > >> Thanks for the suggestion.
>> > >>
>> > >>
>> > >> Regards,
>> > >> Abhimanyu
>> > >>
>> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth909@gmail.com>
>> > wrote:
>> > >>
>> > >>> @Abhimanyu You can try setting offset.retention = 30
>> (log.retention).
>> > At
>> > >>> most, you will have a storage overhead of 5 million msgs per day
*
>> 30
>> > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since
you
>> > have
>> > >>> a
>> > >>> TB of hard disk)
>> > >>>
>> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <kanth909@gmail.com>
>> > wrote:
>> > >>>
>> > >>> > Looking at that ticket and reading the comments it looks like
one
>> of
>> > >>> the
>> > >>> > concern is as follows.
>> > >>> >
>> > >>> > "offsets.retention.minutes is designed to handle the case
that a
>> > >>> consumer
>> > >>> > group goes away forever. In that case, we don't want to store
the
>> > >>> offsets
>> > >>> > for that group forever."
>> > >>> >
>> > >>> > This can simply be addressed by setting offset.retention ==
>> > >>> log.retention
>> > >>> > by default right? In which case offset wont be stored forever
even
>> > when
>> > >>> > consumer group goes away forever. When the consumer group
goes
>> away
>> > >>> forever
>> > >>> > the upper bound to clean up offsets would be equal to
>> log.retention.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <kanth909@gmail.com>
>> > >>> wrote:
>> > >>> >
>> > >>> >> What is your average message size and network speed?
>> > >>> >>
>> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
>> > >>> >> abhimanyunagrath@gmail.com> wrote:
>> > >>> >>
>> > >>> >>> Hi Girish,
>> > >>> >>>
>> > >>> >>> I did not set any value for offsets.retention.minutes
so
>> therefore
>> > >>> what I
>> > >>> >>> think is picking its default value i.e 1440 minutes
so what do
>> you
>> > >>> think
>> > >>> >>> what should I set if I am keeping my data for 30 days?
>> > >>> >>>
>> > >>> >>> Regards,
>> > >>> >>> Abhimanyu
>> > >>> >>>
>> > >>> >>
>> > >>> >>
>> > >>> >
>> > >>>
>> > >>
>> > >>
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message