kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Krahmer <bkrah...@krahmer.com>
Subject Re: Re: Topic discovery when supporting multiple kafka clusters
Date Tue, 06 Dec 2016 20:37:38 GMT
You didn't mention anything about your current configuration, just that 
you are 'out of resources'.  Perhaps you misunderstand how to size your 
partitions per topic, and how partition allocation works.  If your 
brokers are maxed on cpu, and you double the number of brokers but keep 
the replica count the same, I would expect cpu usage to nearly get cut 
in half.  How many brokers do you have, how many topics do you have and 
how many partitions per topic do you have?  What is your resource 
utilization for bandwidth, CPU, and memory?  How many average consumers 
do you have for each topic?


On 06.12.2016 21:23, Yifan Ying wrote:
> Hi Aseem, the concern is to create too many partitions in total in one
> cluster no matter how many brokers I have in this cluster. I think the two
> articles that I mentioned explain why too many partitions in one cluster
> could cause issues.

View raw message