kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrik Kleindl <pklei...@gmail.com>
Subject Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException
Date Tue, 02 Oct 2018 11:54:38 GMT
Hi

We had several incidents where a streams application crashed while
maintaining a global state store.
Updating global state failed. You can restart KafkaStreams to recover from
this error.: org.apache.kafka.clients.consumer.OffsetOutOfRangeException:
Offsets out of range with no configured reset policy for partitions: ...

As we never set this to none I checked the code and found that
StreamsConfig getGlobalConsumerConfigs and getRestoreConsumerConfigs both
set this explicitely:
baseConsumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");

The logs confirms this:
2018-10-02 11:07:06,057 INFO  [org.apache.kafka.common.utils.AppInfoParser]
(ServerService Thread Pool -- 70) - [short-component-name:;
transaction-id:; user-id:; creation-time:]  Kafka version : 2.0.0-cp1
2018-10-02 11:07:06,057 INFO  [org.apache.kafka.common.utils.AppInfoParser]
(ServerService Thread Pool -- 70) - [short-component-name:;
transaction-id:; user-id:; creation-time:]  Kafka commitId :
a8c648ff08b9235d
2018-10-02 11:07:06,104 INFO
[org.apache.kafka.clients.consumer.ConsumerConfig] (ServerService Thread
Pool -- 72) - [short-component-name:; transaction-id:; user-id:;
creation-time:]  ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = none
bootstrap.servers = [...]
check.crcs = true
client.id = ...-3f809a8a-3915-4ae4-9a37-f7a392e3dff3-global-consumer

...

2018-10-02 11:07:06,418 INFO
[org.apache.kafka.streams.processor.internals.StreamThread] (ServerService
Thread Pool -- 72) - [short-component-name:; transaction-id:; user-id:;
creation-time:]  stream-thread
[...-3f809a8a-3915-4ae4-9a37-f7a392e3dff3-StreamThread-1] Creating restore
consumer client
2018-10-02 11:07:06,419 INFO
[org.apache.kafka.clients.consumer.ConsumerConfig] (ServerService Thread
Pool -- 72) - [short-component-name:; transaction-id:; user-id:;
creation-time:]  ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = none
bootstrap.servers = [...]
check.crcs = true
client.id =
...-3f809a8a-3915-4ae4-9a37-f7a392e3dff3-StreamThread-1-restore-consumer

Is this intentional and if yes, why can this not use the default policy and
recover?

best regards

Patrik

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message