kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ehrlich <and...@aehrlich.com>
Subject Re: latency - how to reduce?
Date Mon, 19 Jan 2015 06:51:27 GMT
What instance types are you using in EC2? Are the drives EBS?

On 1/5/15 11:06 PM, Shlomi Hazan wrote:
> Will do. What did you have in mind? just write a big file to disk and
> measure the time it took to write? maybe also read back? using specific
> API's?
> Apart from the local Win machine case, are you aware of any issues with
> Amazon EC2 instances that may be causing that same latency in production?
> Thanks,
> Shlomi
>
> On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <jun@confluent.io> wrote:
>
>> Not setting "log.flush.interval.messages" is good since the default gives
>> the best latency. Could you do some basic I/O testing on the local FS in
>> your windows machine to make sure the I/O latency is ok?
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <shlomi@viber.com> wrote:
>>
>>> Happy new year!
>>> I did not set "log.flush.interval.messages".
>>> I also could not find a default value in the docs.
>>> Could you explain about that?
>>> Thanks,
>>> Shlomi
>>>
>>> On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <jun@confluent.io> wrote:
>>>
>>>> What's your setting of log.flush.interval.messages on the broker?
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>> On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <shlomi@viber.com>
>> wrote:
>>>>> Hi,
>>>>> I am using 0.8.1.1, and I have hundreds of msec latency at best and
>>> even
>>>>> seconds at worst.
>>>>> I have this latency both on production, (with peak load of 30K
>> msg/sec,
>>>>> replication = 2 across 5 brokers, acks = 1),
>>>>> and on the local windows machine using just one process for each of
>>>>> producer, zookeeper, kafka, consumer.
>>>>> Also tried batch.num.messages=1 and producer.type=sync on the local
>>>> machine
>>>>> but saw no improvement.
>>>>> How can I push latency down to several millis, at least when running
>>>> local?
>>>>> Thanks,
>>>>> Shlomi
>>>>>



Mime
View raw message