kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: latency performance test
Date Thu, 16 Jul 2015 23:24:52 GMT
That's a good bet. Running with a few different buffer sizes would verify
this. Adjusting other settings (e.g. max in flight requests, as mentioned
earlier) could also affect latency and throughput.

-Ewen

On Thu, Jul 16, 2015 at 5:20 AM, Yuheng Du <yuheng.du.hust@gmail.com> wrote:

> Hi Ewen,
>
> Thank you for your patient explaining. It is very helpful.
>
> Can we assume that the long latency of ProducerPerformance comes from
> queuing delay in the buffer and it is related to buffer size?
>
> Thank you!
>
> best,
> Yuheng
>
> On Thu, Jul 16, 2015 at 12:21 AM, Ewen Cheslack-Postava <ewen@confluent.io
> >
> wrote:
>
> > The tests are meant to evaluate different things and the way they send
> > messages is the source of the difference.
> >
> > EndToEndLatency works with a single message at a time. It produces the
> > message then waits for the consumer to receive it. This approach
> guarantees
> > there is no delay due to queuing. The goal with this test is to evaluate
> > the *minimum* latency.
> >
> > ProducerPerformance focuses on achieving maximum throughput. This means
> it
> > will enqueue lots of records so it will always have more data to send
> (and
> > can use batching to increase the throughput). Unlike EndToEndLatency,
> this
> > means records may just sit in a queue on the producer for awhile because
> > the maximum number of in flight requests has been reached and it needs to
> > wait for responses for those requests. Since EndToEndLatency only ever
> has
> > one record outstanding, it will never encounter this case.
> >
> > Batching itself doesn't increase the latency because it only occurs when
> > the producer is either a) already unable to send messages anyway or b)
> > linger.ms is greater than 0, but the tests use the default setting that
> > doesn't linger at all.
> >
> > In your example for ProducerPerformance, you have 100 byte records and
> will
> > buffer up to 64MB. Given the batch size of 8K and default producer
> settings
> > of 5 in flight requests, you can roughly think of one round trip time
> > handling 5 * 8K = 40K bytes of data. If your roundtrip is 1ms, then if
> your
> > buffer is full at 64MB it will take you 64 MB / (40 KB/ms) = 1638ms =
> 1.6s.
> > That means that the record that was added at the end of the buffer had to
> > just sit in the buffer for 1.6s before it was sent off to the broker. And
> > if your buffer is consistently full (which it should be for
> > ProducerPerformance since it's sending as fast as it can), that means
> > *every* record waits that long.
> >
> > Of course, these numbers are estimates, depend on my having used 1ms, but
> > hopefully should make it clear why you can see relatively large
> latencies.
> >
> > -Ewen
> >
> >
> > On Wed, Jul 15, 2015 at 1:38 AM, Yuheng Du <yuheng.du.hust@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I have run the end to end latency test and the producerPerformance test
> > on
> > > my kafka cluster according to
> > > https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
> > >
> > > In end to end latency test, the latency was around 2ms. In
> > > producerperformance test, if use batch size 8196 to send 50,000,000
> > > records:
> > >
> > > >bin/kafka-run-class.sh
> > org.apache.kafka.clients.tools.ProducerPerformance
> > > speedx1 50000000 100 -1 acks=1 bootstrap.servers=192.168.1.1:9092
> > > buffer.memory=67108864 batch.size=8196
> > >
> > >
> > > The results show that max latency is 3617ms, avg latency 626.7ms. I
> wanna
> > > know why the latency in producerperformance test is significantly
> larger
> > > than end to end test? Is it because of batching? Are the definitons of
> > > these two latencies different? I looked at the source code and I
> believe
> > > the latency is measure for the producer.send() function to complete. So
> > > does this latency includes transmission delay, transferring delay, and
> > what
> > > other components?
> > >
> > >
> > > Thanks.
> > >
> > >
> > > best,
> > >
> > > Yuheng
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>



-- 
Thanks,
Ewen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message