kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damian Guy <damian....@gmail.com>
Subject Re: Kafka Streams | Impact on rocksdb stores by Rebalancing
Date Tue, 09 Jan 2018 08:57:56 GMT
On Tue, 9 Jan 2018 at 07:42 Sameer Kumar <sam.kum.work@gmail.com> wrote:

> Hi,
>
> I would like to understand how does rebalance affect state stores
> migration. If I have a cluster of 3 nodes, and 1 goes down, the partitions
> for node3 gets assigned to node1 and node2, does the rocksdb on node1/node2
> also starts updating its store from changelog topic.
>
>
Yes the stores will be migrated to node1 and node2 and they will be
restored from the changelog topic


> If yes, then what impact would this migration process have on querying.
>

You can't query the stores until they have all been restored and the
rebalance ends.

>
> Also, if the state store restoration process takes time, how to make sure
> another rebalance doesn''t happen.
>
>
If you don't lose any more nodes then another rebalance won't happen. If
node1 comes back online, then there will be another rebalance, however the
time taken shouldn't be as long as it will already have most of the state
locally, so it only needs to catch up with the remainder of the changelog.
Additionally, you should run with standby tasks. They are updated in the
background and will mean that in the event of failure the other nodes
should already have most of the state locally, so the restoration process
won't take so long


> -Sameer.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message