kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jayesh Thakrar <j_thak...@yahoo.com.INVALID>
Subject Re: latency - how to reduce?
Date Tue, 06 Jan 2015 15:40:28 GMT
Have you tried using the built-in stress test scripts?
bin/kafka-producer-perf-test.sh
bin/kafka-consumer-perf-test.sh

Here's how I stress tested them -
nohup ${KAFKA_HOME}/bin/kafka-producer-perf-test.sh --broker-list ${KAFKA_SERVERS} --topic
${TOPIC_NAME} --new-producer --threads 16 --messages 100000000 1>kafka-producer-perf-test.sh.log
2>&1  &

nohup ${KAFKA_HOME}/bin/kafka-consumer-perf-test.sh --zookeeper ${ZOOKEEPER_QUORUM} --topic
${TOPIC_NAME} --threads 16 1>kafka-consumer-perf-test.sh.log 2>&1  &

And I used screen scrapping of the jmx ui screens to push metrics into TSDB to get the following.The
rate below is per second - so I could push the Kafka cluster to 140k+ messages/sec on a 4-node
cluster with very little utilization (<30% utilization). 


      From: Shlomi Hazan <shlomi@viber.com>
 To: users@kafka.apache.org 
 Sent: Tuesday, January 6, 2015 1:06 AM
 Subject: Re: latency - how to reduce?
   
Will do. What did you have in mind? just write a big file to disk and
measure the time it took to write? maybe also read back? using specific
API's?
Apart from the local Win machine case, are you aware of any issues with
Amazon EC2 instances that may be causing that same latency in production?
Thanks,
Shlomi



On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <jun@confluent.io> wrote:

> Not setting "log.flush.interval.messages" is good since the default gives
> the best latency. Could you do some basic I/O testing on the local FS in
> your windows machine to make sure the I/O latency is ok?
>
> Thanks,
>
> Jun
>
> On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <shlomi@viber.com> wrote:
>
> > Happy new year!
> > I did not set "log.flush.interval.messages".
> > I also could not find a default value in the docs.
> > Could you explain about that?
> > Thanks,
> > Shlomi
> >
> > On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <jun@confluent.io> wrote:
> >
> > > What's your setting of log.flush.interval.messages on the broker?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <shlomi@viber.com>
> wrote:
> > >
> > > > Hi,
> > > > I am using 0.8.1.1, and I have hundreds of msec latency at best and
> > even
> > > > seconds at worst.
> > > > I have this latency both on production, (with peak load of 30K
> msg/sec,
> > > > replication = 2 across 5 brokers, acks = 1),
> > > > and on the local windows machine using just one process for each of
> > > > producer, zookeeper, kafka, consumer.
> > > > Also tried batch.num.messages=1 and producer.type=sync on the local
> > > machine
> > > > but saw no improvement.
> > > > How can I push latency down to several millis, at least when running
> > > local?
> > > > Thanks,
> > > > Shlomi
> > > >
> > >
> >
>


  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message