kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dillian Murphey <crackshotm...@gmail.com>
Subject Re: Poor performance running performance test
Date Wed, 28 Jan 2015 07:21:19 GMT
I was running the performance command from a virtual box server, so that
seems like it was part of the problem.  I'm getting better results running
this on a server on aws, but that's kind of expected.  Can you look at
these results, and comment on the occasional warning I see?  I appreciate
it!

1220375 records sent, 243928.6 records/sec (23.26 MB/sec), 2111.5 ms avg
latency, 4435.0 max latency.
1195090 records sent, 239018.0 records/sec (22.79 MB/sec), 2203.1 ms avg
latency, 4595.0 max latency.
1257165 records sent, 251433.0 records/sec (23.98 MB/sec), 2172.6 ms avg
latency, 4525.0 max latency.
1230981 records sent, 246196.2 records/sec (23.48 MB/sec), 2173.5 ms avg
latency, 4465.0 max latency.
[2015-01-28 07:19:07,274] WARN Error in I/O with
<myawsloadbalancer>(org.apache.kafka.common.network.Selector)
java.io.EOFException
at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62)
at org.apache.kafka.common.network.Selector.poll(Selector.java:248)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122)
at java.lang.Thread.run(Thread.java:745)
1090689 records sent, 218137.8 records/sec (20.80 MB/sec), 2413.6 ms avg
latency, 4829.0 max latency.

On Tue, Jan 27, 2015 at 7:37 PM, Ewen Cheslack-Postava <ewen@confluent.io>
wrote:

> Where are you running ProducerPerformance in relation to ZK and the Kafka
> brokers? You should definitely see much higher performance than this.
>
> A couple of other things I can think of that might be going wrong: Are all
> your VMs in the same AZ? Are you storing Kafka data in EBS or local
> ephemeral storage? If EBS, have you provisioned enough IOPS.
>
>
> On Tue, Jan 27, 2015 at 4:29 PM, Dillian Murphey <crackshotmule@gmail.com>
> wrote:
>
> > I'm a new user/admin to kafka. I'm running a 3 node ZK and a 6 brokers on
> > aws.
> >
> > The performance I'm seeing is shockingly bad. I need some advice!
> >
> > bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
> > test2 50000000 100 -1 acks=1 bootstrap.servers=5<myloadbalancer>:9092
> > buffer.memory=67108864 batch.size=8196
> >
> >
> >
> >
> > 6097 records sent, 13198.3 records/sec (1.26 MB/sec), 2098.0 ms avg
> > latency, 4306.0 max latency.
> > 71695 records sent, 14339.0 records/sec (1.37 MB/sec), 6658.1 ms avg
> > latency, 9053.0 max latency.
> > 65195 records sent, 13028.6 records/sec (1.24 MB/sec), 11504.0 ms avg
> > latency, 13809.0 max latency.
> > 71955 records sent, 14391.0 records/sec (1.37 MB/sec), 16137.4 ms avg
> > latency, 18541.0 max latency.
> >
> > Thanks for any help!
> >
>
>
>
> --
> Thanks,
> Ewen
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message