kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Рябков Алексей Николаевич <a.ryab...@ntc-vulkan.ru>
Subject multi-broker topic rebalance
Date Mon, 26 May 2014 14:06:36 GMT

How can I rebalance topic within multiple brokers?
For example:

1.        I have about 30k consumers...

2.        I need to use 3 broker (10k consumer per broker)

3.       I use multiple (around 10) producers with default balance strategy
Sometimes one broker have empty topic (consumer  finished all message on broker)...But another
broker have a lot of messages in topic...

How to resolve such situation:
-i cant just switch my consumers to another broker ( broker can not handle more than 10k consumers....)
-I can not find how to rebalance topic within multiple brokers...
- I can not find how to setup limit for message per topic (or partition) ... If such things
exists I can handle such case when add new message to broker in consumer side...and switch
to  another ...
Thanks in advance, Aleksey Ryabkov

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message