kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: kafka streams offsets.retention.minutes
Date Fri, 03 Mar 2017 18:59:55 GMT
Hello Tom,

I think there is a general solution proposed to tackle KAFKA-3806, which is
to re-define the semantics of "offset.retention.minutes" such that we will
only start timing it with this config value after the consumer group has
been removed at the coordinator side (i.e. all members are gone for this
group). But I'm not sure if it is agreed upon for this approach. CC'ing
Jason if he can provide some more insights.


On Fri, Mar 3, 2017 at 5:43 AM, Tom Dearman <tom.dearman@gmail.com> wrote:

> I recently realised that our consumers, which don’t use auto commit, will
> suffer from the issue outlined here (https://issues.apache.org/
> jira/browse/KAFKA-3806 <https://issues.apache.org/jira/browse/KAFKA-3806>).
> Looking through the kafka streams code it will also suffer from this
> issue.  I know there is a work around to set the retention time to some
> large value, but given that stream aggregation is likely to be something
> that can happy over a long period of time, and that there might per times
> when a given partition is inactive, would the streams guys consider putting
> in some protection similar to what Jun Rao suggested in the issue ticket?
> Something which will check the last actual commit offset time and override
> commitOffsetNeeded if it hasn’t ben saved within that time.
> Thanks.

-- Guozhang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message