kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException
Date Tue, 02 Oct 2018 20:26:30 GMT
It is by design to set the reset policy to "none"
(https://issues.apache.org/jira/browse/KAFKA-6121), and not allowed by
design to overwrite this (there might be a workaround for you though).
However, Streams should not die but catch the exception and recover from
it automatically.

What version do you use? Can you share the full stack trace to see why
Streams failed to recover from this exception?


-Matthias

On 10/2/18 4:54 AM, Patrik Kleindl wrote:
> Hi
> 
> We had several incidents where a streams application crashed while
> maintaining a global state store.
> Updating global state failed. You can restart KafkaStreams to recover from
> this error.: org.apache.kafka.clients.consumer.OffsetOutOfRangeException:
> Offsets out of range with no configured reset policy for partitions: ...
> 
> As we never set this to none I checked the code and found that
> StreamsConfig getGlobalConsumerConfigs and getRestoreConsumerConfigs both
> set this explicitely:
> baseConsumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");
> 
> The logs confirms this:
> 2018-10-02 11:07:06,057 INFO  [org.apache.kafka.common.utils.AppInfoParser]
> (ServerService Thread Pool -- 70) - [short-component-name:;
> transaction-id:; user-id:; creation-time:]  Kafka version : 2.0.0-cp1
> 2018-10-02 11:07:06,057 INFO  [org.apache.kafka.common.utils.AppInfoParser]
> (ServerService Thread Pool -- 70) - [short-component-name:;
> transaction-id:; user-id:; creation-time:]  Kafka commitId :
> a8c648ff08b9235d
> 2018-10-02 11:07:06,104 INFO
> [org.apache.kafka.clients.consumer.ConsumerConfig] (ServerService Thread
> Pool -- 72) - [short-component-name:; transaction-id:; user-id:;
> creation-time:]  ConsumerConfig values:
> auto.commit.interval.ms = 5000
> auto.offset.reset = none
> bootstrap.servers = [...]
> check.crcs = true
> client.id = ...-3f809a8a-3915-4ae4-9a37-f7a392e3dff3-global-consumer
> 
> ...
> 
> 2018-10-02 11:07:06,418 INFO
> [org.apache.kafka.streams.processor.internals.StreamThread] (ServerService
> Thread Pool -- 72) - [short-component-name:; transaction-id:; user-id:;
> creation-time:]  stream-thread
> [...-3f809a8a-3915-4ae4-9a37-f7a392e3dff3-StreamThread-1] Creating restore
> consumer client
> 2018-10-02 11:07:06,419 INFO
> [org.apache.kafka.clients.consumer.ConsumerConfig] (ServerService Thread
> Pool -- 72) - [short-component-name:; transaction-id:; user-id:;
> creation-time:]  ConsumerConfig values:
> auto.commit.interval.ms = 5000
> auto.offset.reset = none
> bootstrap.servers = [...]
> check.crcs = true
> client.id =
> ...-3f809a8a-3915-4ae4-9a37-f7a392e3dff3-StreamThread-1-restore-consumer
> 
> Is this intentional and if yes, why can this not use the default policy and
> recover?
> 
> best regards
> 
> Patrik
> 


Mime
View raw message