kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Achanta Vamsi Subhash <achanta.va...@flipkart.com>
Subject Re: Kafka Zookeeper queries
Date Tue, 21 Apr 2015 19:00:22 GMT
@Qin
Thanks for the inputs. Could you point us what we should monitor in
zookeeper if we have 20k partitions (this might go ~1 lakh in the coming 6
months)? We are not running zookeeper on SSDs and we were facing
ZkConnectionLoss in HighLevelConsumer, sometimes for all the topics. What
are the critical things to monitor/optimise wrt. zookeeper.

@Pushkar
Yes. We are using kafka for storing offsets instead of zookeeper.

On Tue, Apr 21, 2015 at 11:55 PM, pushkar priyadarshi <
priyadarshi.pushkar@gmail.com> wrote:

> In my knowledge if you are using 0.8.2.1 which is latest stable you can
> sync up your consumer offsets in kafka itself instead of Zk which further
> brings down write load on ZKs.
>
> Regards,
> Pushkar
>
> On Tue, Apr 21, 2015 at 1:13 PM, Jiangjie Qin <jqin@linkedin.com.invalid>
> wrote:
>
> > 20000 partitions should be OK.
> >
> > On 4/21/15, 12:33 AM, "Achanta Vamsi Subhash" <
> achanta.vamsi@flipkart.com>
> > wrote:
> >
> > >We are planning to have ~20000 partitions. Will it be a bottleneck?
> > >
> > >On Mon, Apr 20, 2015 at 10:48 PM, Jiangjie Qin
> <jqin@linkedin.com.invalid
> > >
> > >wrote:
> > >
> > >> Producers usually do not query zookeeper at all.
> > >> Consumers usually query zookeeper at beginning or rebalance. It is
> > >> supposed to be in frequent if you don¹t have consumers come and go all
> > >>the
> > >> time. One exception is that if you are using zookeeper based consumer
> > >> offset commit, it will commit offset to zookeeper frequently.
> > >> In Kafka, the most heavily used mechanism for zookeeper is zookeeper
> > >> listener and they are not fired in a regular frequency.
> > >>
> > >> The limitation of Zookeeper usage for Kafka I am aware of is probably
> > >>the
> > >> size of each zNode. As long as you don¹t have so many partitions that
> > >> zNode cannot handle, it should be fine.
> > >>
> > >> Thanks.
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> > >> On 4/20/15, 5:58 AM, "Achanta Vamsi Subhash"
> > >><achanta.vamsi@flipkart.com>
> > >> wrote:
> > >>
> > >> >Hi,
> > >> >
> > >> >Could anyone help with this?
> > >> >
> > >> >Thanks.
> > >> >
> > >> >On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash <
> > >> >achanta.vamsi@flipkart.com> wrote:
> > >> >
> > >> >> Hi,
> > >> >>
> > >> >> How often does Kafka query zookeeper while producing and consuming?
> > >> >>
> > >> >> Ex:
> > >> >> If there is a single partition to which we produce and a HighLevel
> > >> >> consumer running on it, how many read/write queries to zookeeper
> > >>happen.
> > >> >>
> > >> >> Extending further, multiple topics with ~100 partitions each,
how
> > >>many
> > >> >> zookeeper calls will be made (read/write).
> > >> >>
> > >> >> What is the max limit of no of partitions / kafka cluster that
> > >>zookeeper
> > >> >> can handle?
> > >> >>
> > >> >> --
> > >> >> Regards
> > >> >> Vamsi Subhash
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> >--
> > >> >Regards
> > >> >Vamsi Subhash
> > >>
> > >>
> > >
> > >
> > >--
> > >Regards
> > >Vamsi Subhash
> >
> >
>



-- 
Regards
Vamsi Subhash

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message