kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tony Liu <jiangtao....@zuora.com>
Subject Re: Any suggestion about Kafka/Zookeeper migration.
Date Thu, 04 Jan 2018 04:50:32 GMT
sorry guys.

I attached the picture again.

On Wed, Jan 3, 2018 at 8:49 PM, Tony Liu <jiangtao.liu@zuora.com> wrote:

> for solution A , actually including two parts:
>
>    - zookeeper : manual operation to migrate zookeeper, so far I think it
>    needs at least 6 steps. this part will be tricky, we will have 3 zookeeper
>    exhibitor , and 5 new zookeeper without e
>    - Kafka : this should be easier , try launch new one to join the
>    brokers cluster, and manual migration / reassign partitions.
>
>
> By the way, I also have a concern about `Solution A`, you know we have 5
> production cluster right now, the manual effort will be big and dangerous.
>
> :), that's why I ask your guys experience firstly here.
>
>
> On Wed, Jan 3, 2018 at 8:39 PM, Tony Liu <jiangtao.liu@zuora.com> wrote:
>
>> Hi All,
>>
>> This post here is aimed to ask experience about what did you do migration
>> `Kafka/zookeeper` ?  :)
>>
>> All of Kafka/zookeeper are running on AWS, because of some reason, we
>> have to replace all the existed server (you can simply think we will
>> terminate the old server and create new server to replace it), which is
>> running with Kafka/zookeeper.
>>
>> the Kafka infrastructure :
>>
>>    - 3 zookeeper with exhibitor running together.
>>    - 6 brokers.
>>
>> requirements:
>>
>>    - terminate all of the 3 zookeeper with exhibitor, and also 6 brokers.
>>    - launch 5 new zookeepers without exhibitor.
>>    - launch 6 new brokers.
>>    - 0 or minimum downtime, try our best no impact for the Kafka client.
>>
>>
>> so far, I have two prepared solution, but any of them has pros & cons,
>> the solution A is too much manual operations needed, the solution B is some
>> development effort required and also need to migration all of the Kafka
>> client to use the self-developed Kafka client with `fail-over` feature.
>>
>> *Solution A*: simply speaking, this solution is mainly working on manual
>> operation to add znode / broker to the existed cluster, when all of the
>> thing ready, I will step by step to terminate the old one.
>>
>>
>> ​
>>
>> *Solution B:*  simply speaking, the solution will work like fail-over
>> way, a new cluster will be launched, and control Kafka client's traffic to
>> the new cluster, and terminate the old cluster.
>>
>>
>> ​
>>
>>
>>
>

Mime
View raw message