kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maung Than <maung_t...@apple.com>
Subject Re: Question on output of kafka-producer-perf-test.sh
Date Fri, 30 May 2014 19:07:09 GMT

As you advised, I wrote a script to calculate the file sizes on each broker and run the tests
again. 

We are finding that if we are run this tool ( kafka-producer-perf-test.sh ) while firing from
the producers the data on the broker actually bloomed to almost double.

It only happens with no compression mode. We tested multiple times and it would be nice if
some one can validate this as we can’t think of anything else. 

To make sure each test is equal — we wipe out all the data / metadata directories for all
brokers and zookeeper and start form scratch each time. 

( One topic, five brokers, five partitions, no replication factor, two producer machine with
10 threads each) 


--The performance test command used is : /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
--broker-list vp21q12ic-hpaj020921:9092 --messages 200000000 --topic imessage --threads 20
--message-size 405 --batch-size 1000 --compression-codec 2

I now suspect the TimeOutException with Gzip could be caused by this tool running while we
are sending the data. 

Thanks,
Maung

On May 28, 2014, at 9:37 PM, Jun Rao <junrao@gmail.com> wrote:

> If you start with an empty topic, the log size on the broker gives you the
> total bytes sent.
> 
> Thanks,
> 
> Jun
> 
> 
> On Wed, May 28, 2014 at 10:38 AM, Maung Than <maung_than@apple.com> wrote:
> 
>> 
>>>>>> The bytes-in-rate reported on the broker is the post-compression
rate.
>> 
>> Based on that fact, I am trying to calculate the volume by multiplying
>> duration and rate, i.e,  (end.time - start.time) X MB.sec ,
>> but it does not give me the total.data.sent.in.MB.I put the duration in
>> the brackets below:
>> 
>> Any thoughts?
>> 
>> 
>> 2014-05-27 14:10:37:347,    2014-05-27 14:11:15:546, 0,  420,  1000,
>> 2002.72,   52.4285,  5000000,  130893.4789  ( 34 sec )
>> 
>> 
>> 2014-05-27 13:33:31:493,    2014-05-27 13:34:11:862, 2,  420,  1000,
>> 2002.72,   49.6102,  5000000,  123857.4153  ( 40 sec )
>> 
>> Thanks,
>> Maung
>> 
>> 
>> On May 28, 2014, at 7:37 AM, Jun Rao <junrao@gmail.com> wrote:
>> 
>>> The bytes-in-rate reported on the broker is the post-compression rate.
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> 
>>> On Tue, May 27, 2014 at 9:16 PM, Maung Than <maung_than@apple.com>
>> wrote:
>>> 
>>>> Any idea when and which release that could be included in?
>>>> 
>>>> We would like to have it sooner and can we do something about it?
>>>> 
>>>> Thanks,
>>>> Maung
>>>> 
>>>> On May 27, 2014, at 5:10 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
>>>> 
>>>>> Maung,
>>>>> 
>>>>> Yes, the throughput computed here is based on pre-compression bytes.
In
>>>> the
>>>>> old producer we do not have a metric exposing the compression ratio,
in
>>>> the
>>>>> new producer available in current trunk we can easily add that metric.
>> I
>>>>> have just created a ticket for this a moment ago.
>>>>> 
>>>>> https://issues.apache.org/jira/browse/KAFKA-1472
>>>>> 
>>>>> Guozhang
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, May 27, 2014 at 2:41 PM, Maung Than <maung_than@apple.com>
>>>> wrote:
>>>>> 
>>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> We are seeing total data sent below is the same with or without
>>>>>> compression.
>>>>>> Is it always raw data?
>>>>>> If so is there a way we can get the compressed volume?
>>>>>> 
>>>>>> start.time, end.time, compression, message.size, batch.size,
>>>>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>>>>>> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>>> further
>>>>>> details.
>>>>>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000,
>> 2002.72,
>>>>>> 52.4285, 5000000, 130893.4789
>>>>>> 
>>>>>> 
>>>>>> -bash-4.1$
>>>>>> 
>> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
>>>>>> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic
>>>> imessage
>>>>>> --threads 5 --message-size 420 --batch-size 1000 --compression-codec
2
>>>>>> start.time, end.time, compression, message.size, batch.size,
>>>>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>>>>>> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>>> further
>>>>>> details.
>>>>>> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000,
>> 2002.72,
>>>>>> 49.6102, 5000000, 123857.4153
>>>>>> 
>>>>>> Thanks,
>>>>>> Maung
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> -- Guozhang
>>>> 
>>>> 
>> 
>> 


Mime
View raw message