kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Ariens <dari...@blackberry.com>
Subject RE: Kafka Producer 0.9 performance issue with small messages
Date Mon, 14 Dec 2015 17:50:17 GMT

I was asking last week on the dev list regarding performance in 0.9.x and how best to achieve
optimal message throughput and find your results rather interesting.

Is producing 7142 msg/sec a fairly typical rate for your test environment (I realize you're
 just using your laptop, though).  Are you able to share your peak message rates in your target/production
environment under peak load?

Also--have you noticed any increase/decrease moving from 0.8.x to 0.9.x (if applicable)?

I'm attempting to compare producing and consuming rates using Kafka (0.8.x and 0.9.x) and
our own in-house low overhead library and would be interested in learning how others are faring.

From: Gary Gershon [gary@intermediasciences.com]
Sent: Sunday, December 13, 2015 5:22 PM
To: users@kafka.apache.org
Subject: Re: Kafka Producer 0.9 performance issue with small messages


Yes - you identified the problem!

We had inserted the .get() for debugging, but didn’t think of the (huge!) side-effects.

Using the async callback works perfectly well.

We are now able to send 100,000 records in 14 sec from a laptop to the Bluemix cloud - ~1000x

Thank you very much!


> On Dec 13, 2015, at 2:48 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
> Gary,
> You are calling "kafkaProducer.send(record).get();" for each message, the
> get() call block until the Future is initialized, which effectively
> synchronize all message sent by asking for the ACK for each message before
> sending the next message, hence no batching.
> You can try using "send(record, callback)" for async sending and let the
> callback handle errors from the returned metadata.
> Guozhang
> <snipped>

View raw message