kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raman Gupta <rocketra...@gmail.com>
Subject Streams bug with auto.offset.reset = none?
Date Tue, 09 Jul 2019 16:43:13 GMT
I have a stream that is configured for exactly-once processing. I set
the "auto.offset.reset" of this stream to "none" for two reasons:

1) If an exactly-once stream loses its offset for some reason, using
either of earliest or latest is dangerous: "earliest" implies
reprocessing data, and "latest" may result in skipped messages. Using
"none" allows the ops team to investigate the situation and set the
offsets explicitly.

2) For a topic migration e.g. t-v1 -> t-v2, my exactly once stream
needs to stop reading from t-v1, and start reading from some
particular offset in t-v2 that corresponds to the message after the
last message the previous stream consumed in t-v1. To be safe, I want
my system to fail if the offset for t-v2 has not been set explicitly,
so again, neither "earliest" nor "latest" are an appropriate setting
here.

However, when using "none", even if the offset has been set explicitly
via the application reset tool, the streams client stills throw an
exception on startup: `NoOffsetForPartitionException` for an internal
repartition topic. This seems like a nasty bug / leaky abstraction:
the consumer offsets, and policy on auto.offset.reset for an internal
topic should not be controlled or affected by the user-configured
policy for input topics.

Or am I missing something here?

Regards,
Raman

Mime
View raw message