kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maung Than <maung_t...@apple.com>
Subject Re: Question on output of kafka-producer-perf-test.sh
Date Wed, 28 May 2014 17:38:53 GMT

>>>> The bytes-in-rate reported on the broker is the post-compression rate.

Based on that fact, I am trying to calculate the volume by multiplying duration and rate,
i.e,  (end.time - start.time) X MB.sec , 
but it does not give me the total.data.sent.in.MB.I put the duration in the brackets below:


Any thoughts? 


2014-05-27 14:10:37:347,    2014-05-27 14:11:15:546, 0,  420,  1000,  2002.72,   52.4285,
 5000000,  130893.4789  ( 34 sec )


2014-05-27 13:33:31:493,    2014-05-27 13:34:11:862, 2,  420,  1000,  2002.72,   49.6102,
 5000000,  123857.4153  ( 40 sec ) 

Thanks,
Maung


On May 28, 2014, at 7:37 AM, Jun Rao <junrao@gmail.com> wrote:

> The bytes-in-rate reported on the broker is the post-compression rate.
> 
> Thanks,
> 
> Jun
> 
> 
> On Tue, May 27, 2014 at 9:16 PM, Maung Than <maung_than@apple.com> wrote:
> 
>> Any idea when and which release that could be included in?
>> 
>> We would like to have it sooner and can we do something about it?
>> 
>> Thanks,
>> Maung
>> 
>> On May 27, 2014, at 5:10 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
>> 
>>> Maung,
>>> 
>>> Yes, the throughput computed here is based on pre-compression bytes. In
>> the
>>> old producer we do not have a metric exposing the compression ratio, in
>> the
>>> new producer available in current trunk we can easily add that metric. I
>>> have just created a ticket for this a moment ago.
>>> 
>>> https://issues.apache.org/jira/browse/KAFKA-1472
>>> 
>>> Guozhang
>>> 
>>> 
>>> 
>>> On Tue, May 27, 2014 at 2:41 PM, Maung Than <maung_than@apple.com>
>> wrote:
>>> 
>>>> 
>>>> Hi All,
>>>> 
>>>> We are seeing total data sent below is the same with or without
>>>> compression.
>>>> Is it always raw data?
>>>> If so is there a way we can get the compressed volume?
>>>> 
>>>> start.time, end.time, compression, message.size, batch.size,
>>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>>>> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further
>>>> details.
>>>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72,
>>>> 52.4285, 5000000, 130893.4789
>>>> 
>>>> 
>>>> -bash-4.1$
>>>> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
>>>> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic
>> imessage
>>>> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2
>>>> start.time, end.time, compression, message.size, batch.size,
>>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>>>> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further
>>>> details.
>>>> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72,
>>>> 49.6102, 5000000, 123857.4153
>>>> 
>>>> Thanks,
>>>> Maung
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> -- Guozhang
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message