kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Farmer <m...@frmr.me>
Subject Re: Standby Replicas with In Memory State Stores
Date Wed, 15 Nov 2017 15:57:57 GMT
Yes, in memory stores are backed by a changelog topic as far as I'm aware.
I have filed https://issues.apache.org/jira/browse/KAFKA-6214

On Tue, Nov 14, 2017 at 10:53 PM Matthias J. Sax <matthias@confluent.io>
wrote:

> Thanks for reporting.
>
> Sounds like a bug to me. Please file a Jira.
>
> Question: even if you use an In-Memory store, it's still backed by a
> changelog topic, right?
>
>
> -Matthias
>
> On 11/14/17 3:07 PM, Matt Farmer wrote:
> > Hey everyone,
> >
> > We ran across a little bit of a landmine in Kafka Streams 0.11.0.1.
> >
> > We decided to flip num.standby.replicas to 1 so that we had a hot copy of
> > our state stores on other hosts to facilitate fast failover. However, one
> > of the applications we tried this on currently uses an in memory store.
> > When we flipped the setting on, we then started to restart the running
> > instances.
> >
> > When we restarted the first instance of the application, it came up fine.
> >
> > When we restarted the second instance, both instances crashed with the
> > following exception:
> >
> > java.lang.IllegalStateException: Consumer is not subscribed to any topics
> > or assigned any partitions
> >         at
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1037)
> >         at
> >
> org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:752)
> >         at
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:524)
> >         at
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
> >         at
> >
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
> >
> > Changing the application to use a persistent store seems to have resolved
> > the issue.
> >
> > A few questions:
> >
> > 1. This feels like a bug... is this a bug? :D
> > 2. If so, is this already filed somewhere? I did some searching on JIRA
> but
> > didn't turn up much.
> >
> > Happy to file an issue and (potentially) tinker with a fix if this is
> > indeed a bug.
> >
> > Thanks,
> > Matt Farmer
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message