kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 高 志明 <gz...@hotmail.com>
Subject Does Kafka support federation of clusters?
Date Wed, 06 Apr 2016 14:59:31 GMT
Hi,


For higher availability and more flexible operations (upgrading, rolling ops, etc.), we are
considering to federate two or more  kafka clusters, each with a separated zk cluster. For
example, if "topic-T" has 6 total partitions, 3 on cluster A, and another 3 on cluster B,
messages will be partitioned (by key or random) into all 6 partitions ([A,0-2], [B,0-2]).
On the other side, consumers have to accept two metadata broker lists for both clusters, process
messages and store offsets separately.


Pros:

- more HA

- fewer loss for single broker machine down

- fewer loss when a network switch has problems or a rack is power off

- do green/blue compatible upgrading, and afford crashes of one entire cluster (maybe human
mistakes, maybe bugs)

- with a special library of producers and consumers, we can do incompatible upgrading such
as 0.7 -> 0.8


Cons:

- complex operations

- need wrap a special version of producers and consumers APIs for multiple clusters


Do you think this is a proper solution? Does Kafka have some similar features?



Thanks.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message