kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhimanyu Nagrath <abhimanyunagr...@gmail.com>
Subject Re: Messages are repeating in kafka
Date Mon, 22 May 2017 07:37:18 GMT
Hi Girish ,

Do I need to tune this configuration offsets.retention.check.interval.ms
also . Please let me know if I need to tune any other configuration.


Regards,
Abhimanyu

On Sun, May 21, 2017 at 8:01 PM, Girish Aher <girishaher@gmail.com> wrote:

> Yup, exactly as Kant said.
> Also make sure that the retention of the offsets topic is an upper bound
> across all topics. So in this case, don't create any other topics in the
> future with retention of more than 30 days or otherwise they may have the
> same problem too.
>
> On May 21, 2017 03:25, "Abhimanyu Nagrath" <abhimanyunagrath@gmail.com>
> wrote:
>
>> Hi Kant,
>>
>> Thanks for the suggestion.
>>
>>
>> Regards,
>> Abhimanyu
>>
>> On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth909@gmail.com> wrote:
>>
>>> @Abhimanyu You can try setting offset.retention = 30 (log.retention). At
>>> most, you will have a storage overhead of 5 million msgs per day * 30
>>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you have
>>> a
>>> TB of hard disk)
>>>
>>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <kanth909@gmail.com> wrote:
>>>
>>> > Looking at that ticket and reading the comments it looks like one of
>>> the
>>> > concern is as follows.
>>> >
>>> > "offsets.retention.minutes is designed to handle the case that a
>>> consumer
>>> > group goes away forever. In that case, we don't want to store the
>>> offsets
>>> > for that group forever."
>>> >
>>> > This can simply be addressed by setting offset.retention ==
>>> log.retention
>>> > by default right? In which case offset wont be stored forever even when
>>> > consumer group goes away forever. When the consumer group goes away
>>> forever
>>> > the upper bound to clean up offsets would be equal to log.retention.
>>> >
>>> >
>>> >
>>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <kanth909@gmail.com>
>>> wrote:
>>> >
>>> >> What is your average message size and network speed?
>>> >>
>>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
>>> >> abhimanyunagrath@gmail.com> wrote:
>>> >>
>>> >>> Hi Girish,
>>> >>>
>>> >>> I did not set any value for offsets.retention.minutes so therefore
>>> what I
>>> >>> think is picking its default value i.e 1440 minutes so what do you
>>> think
>>> >>> what should I set if I am keeping my data for 30 days?
>>> >>>
>>> >>> Regards,
>>> >>> Abhimanyu
>>> >>>
>>> >>
>>> >>
>>> >
>>>
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message