kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Standby Replicas with In Memory State Stores
Date Wed, 15 Nov 2017 19:26:18 GMT
Thanks!

On 11/15/17 7:57 AM, Matt Farmer wrote:
> Yes, in memory stores are backed by a changelog topic as far as I'm aware.
> I have filed https://issues.apache.org/jira/browse/KAFKA-6214
> 
> On Tue, Nov 14, 2017 at 10:53 PM Matthias J. Sax <matthias@confluent.io>
> wrote:
> 
>> Thanks for reporting.
>>
>> Sounds like a bug to me. Please file a Jira.
>>
>> Question: even if you use an In-Memory store, it's still backed by a
>> changelog topic, right?
>>
>>
>> -Matthias
>>
>> On 11/14/17 3:07 PM, Matt Farmer wrote:
>>> Hey everyone,
>>>
>>> We ran across a little bit of a landmine in Kafka Streams 0.11.0.1.
>>>
>>> We decided to flip num.standby.replicas to 1 so that we had a hot copy of
>>> our state stores on other hosts to facilitate fast failover. However, one
>>> of the applications we tried this on currently uses an in memory store.
>>> When we flipped the setting on, we then started to restart the running
>>> instances.
>>>
>>> When we restarted the first instance of the application, it came up fine.
>>>
>>> When we restarted the second instance, both instances crashed with the
>>> following exception:
>>>
>>> java.lang.IllegalStateException: Consumer is not subscribed to any topics
>>> or assigned any partitions
>>>         at
>>>
>> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1037)
>>>         at
>>>
>> org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:752)
>>>         at
>>>
>> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:524)
>>>         at
>>>
>> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
>>>         at
>>>
>> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
>>>
>>> Changing the application to use a persistent store seems to have resolved
>>> the issue.
>>>
>>> A few questions:
>>>
>>> 1. This feels like a bug... is this a bug? :D
>>> 2. If so, is this already filed somewhere? I did some searching on JIRA
>> but
>>> didn't turn up much.
>>>
>>> Happy to file an issue and (potentially) tinker with a fix if this is
>>> indeed a bug.
>>>
>>> Thanks,
>>> Matt Farmer
>>>
>>
>>
> 


Mime
View raw message