kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Chan ...@ooyala.com>
Subject Kafka performance numbers on EC2
Date Fri, 03 Feb 2012 00:04:12 GMT
Hi folks,

I did a performance test on EC2 using one Kafka box on an m1.large, and
separate m1.large boxes for the producer-perf-shell and the
consumer-perf-shell.
I'm using Kafka 0.6 with no mirroring.
With the message size set to 1000 bytes, here is what I found:

   - Writing Kafka messages to /mnt (EBS), we can write 21,190 messages per
   second.
   - If we write Kafka messages to /tmp (local disk), it's only slightly
   faster at 21,944 messages / sec
   - The number of partitions doesn't seem to have a bearing on
   write/consumer performance

It's pretty interesting that writing to /mnt or EBS doesn't seem to really
hurt the bandwidth.

I also found that the kafka-consumer-perf-test seems to misreport the
"nMsgs/sec".  It goes up from a small number at the beginning of the test
to a large number, which tells me that there is some elapsed / original
timestamp that was set waaay before the first message was consumed.  Maybe
this is fixed in 0.7?

I'd be interested to compare notes with others who have deployed Kafka to
EC2.

thanks,
Evan
-- 
--
*Evan Chan*
Senior Software Engineer |
ev@ooyala.com | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message