kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tony Liu <jiangtao....@zuora.com>
Subject Re: Any suggestion about Kafka/Zookeeper migration.
Date Thu, 04 Jan 2018 04:49:05 GMT
for solution A , actually including two parts:

   - zookeeper : manual operation to migrate zookeeper, so far I think it
   needs at least 6 steps. this part will be tricky, we will have 3 zookeeper
   exhibitor , and 5 new zookeeper without e
   - Kafka : this should be easier , try launch new one to join the brokers
   cluster, and manual migration / reassign partitions.


By the way, I also have a concern about `Solution A`, you know we have 5
production cluster right now, the manual effort will be big and dangerous.

:), that's why I ask your guys experience firstly here.


On Wed, Jan 3, 2018 at 8:39 PM, Tony Liu <jiangtao.liu@zuora.com> wrote:

> Hi All,
>
> This post here is aimed to ask experience about what did you do migration
> `Kafka/zookeeper` ?  :)
>
> All of Kafka/zookeeper are running on AWS, because of some reason, we have
> to replace all the existed server (you can simply think we will terminate
> the old server and create new server to replace it), which is running with
> Kafka/zookeeper.
>
> the Kafka infrastructure :
>
>    - 3 zookeeper with exhibitor running together.
>    - 6 brokers.
>
> requirements:
>
>    - terminate all of the 3 zookeeper with exhibitor, and also 6 brokers.
>    - launch 5 new zookeepers without exhibitor.
>    - launch 6 new brokers.
>    - 0 or minimum downtime, try our best no impact for the Kafka client.
>
>
> so far, I have two prepared solution, but any of them has pros & cons, the
> solution A is too much manual operations needed, the solution B is some
> development effort required and also need to migration all of the Kafka
> client to use the self-developed Kafka client with `fail-over` feature.
>
> *Solution A*: simply speaking, this solution is mainly working on manual
> operation to add znode / broker to the existed cluster, when all of the
> thing ready, I will step by step to terminate the old one.
>
>
> ​
>
> *Solution B:*  simply speaking, the solution will work like fail-over
> way, a new cluster will be launched, and control Kafka client's traffic to
> the new cluster, and terminate the old cluster.
>
>
> ​
>
>
>

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message