kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiangjie Qin <j...@linkedin.com.INVALID>
Subject Re: How many messages does each broker have?
Date Wed, 26 Nov 2014 01:07:52 GMT

Just adding to what Guozhang said, the answer to your question might
depend on which producer you are using.
Assuming you are producing messages without keys to the same topic, in new
producer(KafkaProducer), the messages will go to brokers in a round robin
way, so the messages will end up in brokers evenly distributed. Whereas in
old producer, it actually sticks to a particular broker for 10 min (by
default) then switch to another random partition. In that case, if you
send messages fast enough, you might see uneven distribution in brokers.

For the consumer, if you are using high level consumer, when reading from
KafkaStream, you will get MessageAndMetadata, the topic and partition
information is included in it as well as the raw message.

Jiangjie (Becket) Qin

On 11/25/14, 10:01 AM, "Guozhang Wang" <wangguoz@gmail.com> wrote:

>If the 8 partitions are hosted on each one of the nodes, assuming
>replication factor 1 then each node will get roughly 100000 / 8 messages
>due to the random partitioner. If you want to know exactly how many
>messages is on each broker then you can use a simple consumer which allows
>you to specify the partition id you want to consume from.
>In the new consumer (0.9), each of the consumed message will contain the
>partition id as part of its message metadata.
>On Tue, Nov 25, 2014 at 7:47 AM, Palur Sandeep <psandeep@hawk.iit.edu>
>> Dear Developers,
>> I am using the default partitioning logic(Random Partitioning) to
>> messages into brokers. That is I don't use a partitioner.class.
>> My requirement is If I produce 100000 messages using the below code for
>> broker that has 8 partitions across 8 nodes. How many messages will each
>> partition have? Is there any API that can help me find the broker id of
>> each message I consume from the consumer side?
>> PS: I dont want to use partitioner.class. I want use the kafka's default
>> partitioning logic.
>>   KeyedMessage<String,String> data = new KeyedMessage<String,
>> String>(topic_name,new_mes);
>> producer.send(data);
>> --
>> Regards,
>> Sandeep Palur
>> Data-Intensive Distributed Systems Laboratory, CS/IIT
>> Department of Computer Science, Illinois Institute of Technology (IIT)
>> Phone : 312-647-9833
>> Email : psandeep@hawk.iit.edu <srajasur@hawk.iit.edu>
>-- Guozhang

View raw message