kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiangjie Qin <j...@linkedin.com.INVALID>
Subject Re: Kafka Zookeeper queries
Date Wed, 22 Apr 2015 00:17:18 GMT
I might not be the best one to answer this question. It will be better if
someone has more operation experience can help out here.

Jiangjie (Becket) Qin

On 4/21/15, 12:00 PM, "Achanta Vamsi Subhash" <achanta.vamsi@flipkart.com>
wrote:

>@Qin
>Thanks for the inputs. Could you point us what we should monitor in
>zookeeper if we have 20k partitions (this might go ~1 lakh in the coming 6
>months)? We are not running zookeeper on SSDs and we were facing
>ZkConnectionLoss in HighLevelConsumer, sometimes for all the topics. What
>are the critical things to monitor/optimise wrt. zookeeper.
>
>@Pushkar
>Yes. We are using kafka for storing offsets instead of zookeeper.
>
>On Tue, Apr 21, 2015 at 11:55 PM, pushkar priyadarshi <
>priyadarshi.pushkar@gmail.com> wrote:
>
>> In my knowledge if you are using 0.8.2.1 which is latest stable you can
>> sync up your consumer offsets in kafka itself instead of Zk which
>>further
>> brings down write load on ZKs.
>>
>> Regards,
>> Pushkar
>>
>> On Tue, Apr 21, 2015 at 1:13 PM, Jiangjie Qin
>><jqin@linkedin.com.invalid>
>> wrote:
>>
>> > 20000 partitions should be OK.
>> >
>> > On 4/21/15, 12:33 AM, "Achanta Vamsi Subhash" <
>> achanta.vamsi@flipkart.com>
>> > wrote:
>> >
>> > >We are planning to have ~20000 partitions. Will it be a bottleneck?
>> > >
>> > >On Mon, Apr 20, 2015 at 10:48 PM, Jiangjie Qin
>> <jqin@linkedin.com.invalid
>> > >
>> > >wrote:
>> > >
>> > >> Producers usually do not query zookeeper at all.
>> > >> Consumers usually query zookeeper at beginning or rebalance. It is
>> > >> supposed to be in frequent if you don¹t have consumers come and go
>>all
>> > >>the
>> > >> time. One exception is that if you are using zookeeper based
>>consumer
>> > >> offset commit, it will commit offset to zookeeper frequently.
>> > >> In Kafka, the most heavily used mechanism for zookeeper is
>>zookeeper
>> > >> listener and they are not fired in a regular frequency.
>> > >>
>> > >> The limitation of Zookeeper usage for Kafka I am aware of is
>>probably
>> > >>the
>> > >> size of each zNode. As long as you don¹t have so many partitions
>>that
>> > >> zNode cannot handle, it should be fine.
>> > >>
>> > >> Thanks.
>> > >>
>> > >> Jiangjie (Becket) Qin
>> > >>
>> > >> On 4/20/15, 5:58 AM, "Achanta Vamsi Subhash"
>> > >><achanta.vamsi@flipkart.com>
>> > >> wrote:
>> > >>
>> > >> >Hi,
>> > >> >
>> > >> >Could anyone help with this?
>> > >> >
>> > >> >Thanks.
>> > >> >
>> > >> >On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash <
>> > >> >achanta.vamsi@flipkart.com> wrote:
>> > >> >
>> > >> >> Hi,
>> > >> >>
>> > >> >> How often does Kafka query zookeeper while producing and
>>consuming?
>> > >> >>
>> > >> >> Ex:
>> > >> >> If there is a single partition to which we produce and a
>>HighLevel
>> > >> >> consumer running on it, how many read/write queries to zookeeper
>> > >>happen.
>> > >> >>
>> > >> >> Extending further, multiple topics with ~100 partitions each,
>>how
>> > >>many
>> > >> >> zookeeper calls will be made (read/write).
>> > >> >>
>> > >> >> What is the max limit of no of partitions / kafka cluster
that
>> > >>zookeeper
>> > >> >> can handle?
>> > >> >>
>> > >> >> --
>> > >> >> Regards
>> > >> >> Vamsi Subhash
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> >--
>> > >> >Regards
>> > >> >Vamsi Subhash
>> > >>
>> > >>
>> > >
>> > >
>> > >--
>> > >Regards
>> > >Vamsi Subhash
>> >
>> >
>>
>
>
>
>-- 
>Regards
>Vamsi Subhash

Mime
View raw message