kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitay Kufert <nita...@ironsrc.com>
Subject Consumer loses partition offset and resets post 2.4.1 version upgrade
Date Sun, 05 Apr 2020 17:13:25 GMT
>
> Hey,

We are using kafka streams across our tech stack for the last 2-3 years.

Lately, we moved to kafka 2.4.1 and we started noticing a troubling
behaviour that we didn't see before:

Without apparent reason, a specific partition on a specific consumer loses
its offset, with the following messages appearing on the consumer:

>
> [Consumer clientId=consumer-XConsumerGroup4-2, groupId=XConsumerGroup4]
> Fetch offset 1222791071 is out of range for partition SomeTopic-48,
> resetting offset:
>
 [Consumer clientId=consumer-XConsumerGroup4-2, groupId=XConsumerGroup4]
> Resetting offset for partition SomeTopic-48 to offset 1216430527.:


from the broker, the only thing i could find is:

> BrokerId 1033 is no longer a coordinator for the group XConsumerGroup4. Proceeding cleanup
for other alive groups
>
>
Another way to see the same thing, from our monitoring (DD):
[image: Screen Shot 2020-04-05 at 16.32.19.png]

The recovery you are seeing is after I run partition offset reset manually
(using kafka-consumer-groups.sh --bootstrap-server localhost:9092 --topic
SomeTopic:57 --group XConsumerGroup4 --reset-offsets --to-datetime
'2020-04-02T07:00:00.000')

Any idea what can be causing this? we have it happen to us at least 5 times
since the upgrade, and before that i don't remember it even happening to us.

Let me know if you need more data from me
Thanks
-- 

Nitay Kufert
Backend Team Leader
[image: ironSource] <http://www.ironsrc.com>

email nitay.k@ironsrc.com
mobile +972-54-5480021
fax +972-77-5448273
skype nitay.kufert.ssa
121 Menachem Begin St., Tel Aviv, Israel
ironsrc.com <http://www.ironsrc.com>
[image: linkedin] <https://www.linkedin.com/company/ironsource> [image:
twitter] <https://twitter.com/ironsource> [image: facebook]
<https://www.facebook.com/ironSource> [image: googleplus]
<https://plus.google.com/+ironsrc>
This email (including any attachments) is for the sole use of the intended
recipient and may contain confidential information which may be protected
by legal privilege. If you are not the intended recipient, or the employee
or agent responsible for delivering it to the intended recipient, you are
hereby notified that any use, dissemination, distribution or copying of
this communication and/or its content is strictly prohibited. If you are
not the intended recipient, please immediately notify us by reply email or
by telephone, delete this email and destroy any copies. Thank you.

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message