kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Dearman <tom.dear...@gmail.com>
Subject kafka streams offsets.retention.minutes
Date Fri, 03 Mar 2017 13:43:53 GMT
I recently realised that our consumers, which don’t use auto commit, will suffer from the
issue outlined here (https://issues.apache.org/jira/browse/KAFKA-3806 <https://issues.apache.org/jira/browse/KAFKA-3806>).
 Looking through the kafka streams code it will also suffer from this issue.  I know there
is a work around to set the retention time to some large value, but given that stream aggregation
is likely to be something that can happy over a long period of time, and that there might
per times when a given partition is inactive, would the streams guys consider putting in some
protection similar to what Jun Rao suggested in the issue ticket?  Something which will check
the last actual commit offset time and override commitOffsetNeeded if it hasn’t ben saved
within that time.

Thanks.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message