kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Widman <j...@jeffwidman.com>
Subject Re: Number of partition on a single broker
Date Thu, 14 Dec 2017 16:56:14 GMT
I have also seen slower replication across the cluster when partitions per
broker are abnormally high, even though the bytes/message throughput isn't
that high. Due to legacy reasons, we have a lot of partitions per broker,
with only a handful really hot and the others just barely trickling data,
or even completely unused. And I can clearly see that it takes a while for
the brokers to service those replication requests. There's just more
metadata management that the brokers have to do internally, even when the
extra partitions are unused. The limit appears to be on a per-broker basis,
I suspect if we added more brokers the cluster would be just fine.

On Dec 14, 2017 6:00 AM, "Todd Palino" <tpalino@gmail.com> wrote:

Not recommended. You’ll have timeout issues with the size of the controller
requests. Additionally, there appear to be problems with writing some nodes
in Zookeeper at high partition counts.

-Todd


On Thu, Dec 14, 2017 at 8:58 AM, Abhimanyu Nagrath <
abhimanyunagrath@gmail.com> wrote:

> Can I have 20k partition on a single kafka broker ?
>



--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming



linkedin.com/in/toddpalino

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message