kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismael Juma <ism...@juma.me.uk>
Subject Re: Consumer loses partition offset and resets post 2.4.1 version upgrade
Date Sun, 05 Apr 2020 17:51:54 GMT
Hi Nitay,

What version were you running before the upgrade? And was this a broker
upgrade? I would suggest filing a JIRA ticket, including as much
information as possible and ideally consumer and broker logs from the
period where the issue happened.

Ismael

On Sun, Apr 5, 2020, 10:13 AM Nitay Kufert <nitay.k@ironsrc.com> wrote:

> Hey,
>
> We are using kafka streams across our tech stack for the last 2-3 years.
>
> Lately, we moved to kafka 2.4.1 and we started noticing a troubling
> behaviour that we didn't see before:
>
> Without apparent reason, a specific partition on a specific consumer loses
> its offset, with the following messages appearing on the consumer:
>
>>
>> [Consumer clientId=consumer-XConsumerGroup4-2, groupId=XConsumerGroup4]
>> Fetch offset 1222791071 is out of range for partition SomeTopic-48,
>> resetting offset:
>>
>  [Consumer clientId=consumer-XConsumerGroup4-2, groupId=XConsumerGroup4]
>> Resetting offset for partition SomeTopic-48 to offset 1216430527.:
>
>
> from the broker, the only thing i could find is:
>
>> BrokerId 1033 is no longer a coordinator for the group XConsumerGroup4. Proceeding
cleanup for other alive groups
>>
>>
> Another way to see the same thing, from our monitoring (DD):
> [image: Screen Shot 2020-04-05 at 16.32.19.png]
>
> The recovery you are seeing is after I run partition offset reset manually
> (using kafka-consumer-groups.sh --bootstrap-server localhost:9092 --topic
> SomeTopic:57 --group XConsumerGroup4 --reset-offsets --to-datetime
> '2020-04-02T07:00:00.000')
>
> Any idea what can be causing this? we have it happen to us at least 5
> times since the upgrade, and before that i don't remember it even happening
> to us.
>
> Let me know if you need more data from me
> Thanks
> --
>
> Nitay Kufert
> Backend Team Leader
> [image: ironSource] <http://www.ironsrc.com>
>
> email nitay.k@ironsrc.com
> mobile +972-54-5480021
> fax +972-77-5448273
> skype nitay.kufert.ssa
> 121 Menachem Begin St., Tel Aviv, Israel
> ironsrc.com <http://www.ironsrc.com>
> [image: linkedin] <https://www.linkedin.com/company/ironsource> [image:
> twitter] <https://twitter.com/ironsource> [image: facebook]
> <https://www.facebook.com/ironSource> [image: googleplus]
> <https://plus.google.com/+ironsrc>
> This email (including any attachments) is for the sole use of the intended
> recipient and may contain confidential information which may be protected
> by legal privilege. If you are not the intended recipient, or the employee
> or agent responsible for delivering it to the intended recipient, you are
> hereby notified that any use, dissemination, distribution or copying of
> this communication and/or its content is strictly prohibited. If you are
> not the intended recipient, please immediately notify us by reply email or
> by telephone, delete this email and destroy any copies. Thank you.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message