kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Achanta Vamsi Subhash <achanta.va...@flipkart.com>
Subject Re: Kafka Zookeeper queries
Date Tue, 21 Apr 2015 07:33:43 GMT
We are planning to have ~20000 partitions. Will it be a bottleneck?

On Mon, Apr 20, 2015 at 10:48 PM, Jiangjie Qin <jqin@linkedin.com.invalid>
wrote:

> Producers usually do not query zookeeper at all.
> Consumers usually query zookeeper at beginning or rebalance. It is
> supposed to be in frequent if you don¹t have consumers come and go all the
> time. One exception is that if you are using zookeeper based consumer
> offset commit, it will commit offset to zookeeper frequently.
> In Kafka, the most heavily used mechanism for zookeeper is zookeeper
> listener and they are not fired in a regular frequency.
>
> The limitation of Zookeeper usage for Kafka I am aware of is probably the
> size of each zNode. As long as you don¹t have so many partitions that
> zNode cannot handle, it should be fine.
>
> Thanks.
>
> Jiangjie (Becket) Qin
>
> On 4/20/15, 5:58 AM, "Achanta Vamsi Subhash" <achanta.vamsi@flipkart.com>
> wrote:
>
> >Hi,
> >
> >Could anyone help with this?
> >
> >Thanks.
> >
> >On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash <
> >achanta.vamsi@flipkart.com> wrote:
> >
> >> Hi,
> >>
> >> How often does Kafka query zookeeper while producing and consuming?
> >>
> >> Ex:
> >> If there is a single partition to which we produce and a HighLevel
> >> consumer running on it, how many read/write queries to zookeeper happen.
> >>
> >> Extending further, multiple topics with ~100 partitions each, how many
> >> zookeeper calls will be made (read/write).
> >>
> >> What is the max limit of no of partitions / kafka cluster that zookeeper
> >> can handle?
> >>
> >> --
> >> Regards
> >> Vamsi Subhash
> >>
> >
> >
> >
> >--
> >Regards
> >Vamsi Subhash
>
>


-- 
Regards
Vamsi Subhash

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message