kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Chu <david....@appdynamics.com>
Subject Possible to Disable Offset Checkpointing for an In-Memory Global Store?
Date Tue, 17 Apr 2018 21:16:34 GMT
I have a custom in-memory state store which I’d like to configure as a global store for my
Kafka Streams application.  However, even though my state store is not persistent (i.e. the
implementation of StateStore.persistent() returns false) it appears that the GlobalStateManager
still continues to write offset checkpoints to disk and then reads those checkpoints upon
recovery of the Kafka Streams application to adjust the offsets of the globalConsumer.  Because
my state store is not persistent though, it does not preserve the previous data which was
read from the topic before the streams application was restarted.  Therefore, the fact that
the globalConsumer has its offsets set to the values read from the checkpoint file upon initialization
causes my state store to lose previously consumed data.  To avoid this situation, it would
seem like non-persistent state stores should not have their offsets checkpointed when they
are used as a global store; this way the globalConsumer would always consume events from the
beginning of the topic.  This has the consequence of slowing down start-up time of the stream
since it will have to recover more messages from the topic but in my case the number of messages
on this topic is not expected to be large and I’m willing to tolerate the slightly slower
start-up times.  Therefore, is it possible to disable offset checkpointing for in-memory global
stores?  If not, is it reasonable to request such a feature to be added?

Thanks,
David 
Mime
View raw message