kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Bejeck <b...@confluent.io>
Subject Re: Kafka Streams - Custom processor "init" method called before state store has data restored into it
Date Thu, 09 Nov 2017 18:43:19 GMT
Hi Rainer,

Thanks for reporting this issue.  Do you have any log data you can share?

In the meantime, I'll look into the issue.

Thanks,
Bill

On Thu, Nov 9, 2017 at 1:23 PM, Rainer Guessner <raguessner@gmx.com> wrote:

> I have a custom processor that implements AbstractProcessor and a custom
> store that implements StateStore.
>
> Before Kafka 1.0.0 the processors "init" method gets called after the
> state store is restored from changelog and that is good.
> With Kafka 1.0.0 the processors "init" method is called BEFORE the state
> store is restored from changelog and that is bad.
>
> My processor can only initialize when it has access to the state. However
> at the time KStreams calls "init" on the processor the state store may not
> have any data. It is not an option for me to initialize the processor
> lazily when a record arrives, or to re-initialize it when "onRestoreEnd" is
> called (its only called on restore; The state store "init" gets called
> before processor "init" regardless of restore or not.)
>
> I think I need to have either of these:
> a) know whether or not a state restore will take place and when not
> b) or get a call to the state store regardless of whether state restore
> took place or not
> c) or I need a "ready" method on the processor that gets called when the
> state store has completed restoring and is actually usable
>
> Please help, thank you in advance.
> Rainer
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message