kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Akhtar <ali.rac...@gmail.com>
Subject Re: hi
Date Thu, 15 Sep 2016 08:47:21 GMT
The issue is clearly that you're running out of resources, so I would add
more brokers and/or larger instances.

You're also using Node which is not the best for performance. A compiled
language such as Java would give you the best performance.

Here's a case study that should help:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

Good luck, let us know how it goes

On Thu, Sep 15, 2016 at 1:42 PM, kant kodali <kanth909@gmail.com> wrote:

> yeah..
> I tried it with 10 messages with single broker and only one partiton that
> looked
> instantaneous and ~5K messages/sec for the data size of 1KBI tried it with
> 1000
> messages that looked instantaneous as well ~5K messages/sec for the data
> size of
> 1KBI tried it with 10K messages with single broker and only one
> partiton  things
> started to go down ~1K messages/sec for the data size of 1KB
> having only one partition on a single broker is a bad?  My goal is to run
> some
> basic benchmarks on NATS & NSQ & KAFKA
> I have the same environment for all three (NATS & NSQ & KAFKA)
> a broker  on Machine 1producer on Machine 2Consumer on Machine 3
> with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
> instance.
> I have pushed 300K messages with NATS and it was able to handle easily and
> receive throughput was 5K messages/secI have pushed 300K messages and NSQ
> and
> receive throughput was 2K messages/secI am unable to push 300K messages
> with
> Kafka with the above configuration and environment so at this point my
> biggest
> question is what is the fair setup for Kafka so its comparable with NATS
> and
> NSQ?
> kant
>
>
>
>
>
>
> On Thu, Sep 15, 2016 12:43 AM, Ali Akhtar ali.rac200@gmail.com
> wrote:
> Lower the workload gradually, start from 10 messages, increase to 100, then
>
> 1000, and so on. See if it slows down as the workload increases. If so, you
>
> need more brokers + partitions to handle the workload.
>
>
>
>
> On Thu, Sep 15, 2016 at 12:42 PM, kant kodali <kanth909@gmail.com> wrote:
>
>
>
>
> > m4.xlarge
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac200@gmail.com
>
> >
>
> > wrote:
>
> > What's the instance size that you're using? With 300k messages your
> single
>
> >
>
> > broker might not be able to handle it.
>
> >
>
> >
>
> >
>
> >
>
> > On Thu, Sep 15, 2016 at 12:30 PM, kant kodali <kanth909@gmail.com>
> wrote:
>
> >
>
> >
>
> >
>
> >
>
> > My goal is to test the throughput (#messages per second) given my setup
> and
>
> >>
>
> >
>
> > with a data size of 1KB. if you guys already have some idea on these
>
> >>
>
> >
>
> > numbers
>
> >>
>
> >
>
> > that would be helpful as well.
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth909@gmail.com
>
> >>
>
> >
>
> > wrote:
>
> >>
>
> >
>
> > 172.* is all private ip's for my machine I double checked it.I have not
>
> >>
>
> >
>
> > changed
>
> >>
>
> >
>
> > any default settingsI dont know how to use kafka-consumer.sh
>
> >>
>
> >
>
> > or kafka-producer.sh because it looks like they want me to specify a
> group
>
> >>
>
> >
>
> > and I
>
> >>
>
> >
>
> > didn't create any consumer group because I am using single producer and
>
> >>
>
> >
>
> > consumer. is there a default group?Also, I am receiving message but very
>
> >>
>
> >
>
> > late. I
>
> >>
>
> >
>
> > send about 300K messages using the node.js client and I am receiving at a
>
> >>
>
> >
>
> > very
>
> >>
>
> >
>
> > low rate. really not sure what is going on?
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac200@gmail.com
>
> >>
>
> >
>
> > wrote:
>
> >>
>
> >
>
> > Your code seems to be using the public ip of the servers. If all 3
> machines
>
> >>
>
> >
>
> >
>
> >>
>
> > are in the same availability zone on AWS, try using the private ip, and
>
> >>
>
> >
>
> >
>
> >>
>
> > then they might communicate over the local network.
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > Did you change any default settings?
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > Do you get the same results if you run kafka-consumer.sh and
>
> >>
>
> >
>
> >
>
> >>
>
> > kafka-producer.sh instead of the Node code?
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > On Thu, Sep 15, 2016 at 12:01 PM, kant kodali <kanth909@gmail.com>
> wrote:
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > > They are hosted on AWS and I dont think there are any network issues
>
> >>
>
> >
>
> >
>
> >>
>
> > > because I
>
> >>
>
> >
>
> >
>
> >>
>
> > > tried testing other Queuing systems with no issues however I am using a
>
> >>
>
> >
>
> >
>
> >>
>
> > > node.js
>
> >>
>
> >
>
> >
>
> >>
>
> > > client with the following code. I am not sure if there are any errors
> or
>
> >>
>
> >
>
> >
>
> >>
>
> > > anything I didn't set in the following code?
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > //producer var kafka = require('kafka-node'); var
>
> >>
>
> >
>
> >
>
> >>
>
> > > Producer = kafka.Producer; var Client = kafka.Client; var client =
>
> >>
>
> >
>
> >
>
> >>
>
> > > new Client('172.31.21.175:2181'); var argv =
>
> >>
>
> >
>
> >
>
> >>
>
> > > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var
>
> >>
>
> >
>
> >
>
> >>
>
> > > p = argv.p || 0; var a = argv.a || 0; var producer = new
>
> >>
>
> >
>
> >
>
> >>
>
> > > Producer(client, { requireAcks: 1}); var num = 300005;
>
> >>
>
> >
>
> >
>
> >>
>
> > > producer.on('ready', function () { var message = 'Hello World';
>
> >>
>
> >
>
> >
>
> >>
>
> > > for (var i=0; i<num; i++) { producer.send([ { topic:
>
> >>
>
> >
>
> >
>
> >>
>
> > > topic, partition: p, messages: message, attributes: a } ], function
>
> >>
>
> >
>
> >
>
> >>
>
> > > (err, result) { console.log(err || result);
>
> >>
>
> >
>
> >
>
> >>
>
> > > //process.exit(); }); } }); producer.on('error',
>
> >>
>
> >
>
> >
>
> >>
>
> > > function (err) { console.log('error', err); process.exit();
>
> >>
>
> >
>
> >
>
> >>
>
> > > }); //Consumer var kafka = require('kafka-node'); var Consumer =
>
> >>
>
> >
>
> >
>
> >>
>
> > > kafka.Consumer; var Offset = kafka.Offset; var Client =
>
> >>
>
> >
>
> >
>
> >>
>
> > > kafka.Client; var argv = require('optimist').argv; var topic =
>
> >>
>
> >
>
> >
>
> >>
>
> > > argv.topic || 'kafka_test'; var client = new
>
> >>
>
> >
>
> >
>
> >>
>
> > > Client('172.31.21.175:2181'); var topics = [ {topic: topic,
>
> >>
>
> >
>
> >
>
> >>
>
> > > partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:
>
> >>
>
> >
>
> >
>
> >>
>
> > > 1000 }; var consumer = new Consumer(client, topics, options); var
>
> >>
>
> >
>
> >
>
> >>
>
> > > offset = new Offset(client); var start; var received = 0; var
>
> >>
>
> >
>
> >
>
> >>
>
> > > target = 200000; var hash = 1000; consumer.on('message', function
>
> >>
>
> >
>
> >
>
> >>
>
> > > (message) { console.log(message); received += 1; if
>
> >>
>
> >
>
> >
>
> >>
>
> > > (received === 1) { start = new Date(); } if (received === target) {
>
> >>
>
> >
>
> >
>
> >>
>
> > > var stop = new Date(); console.log('\nDone test');
>
> >>
>
> >
>
> >
>
> >>
>
> > > var mps = parseInt(target/((stop-start)/1000));
>
> >>
>
> >
>
> >
>
> >>
>
> > > console.log('Received at ' + mps + ' msgs/sec'); process.exit();
>
> >>
>
> >
>
> >
>
> >>
>
> > > } else if (received % hash === 0){
>
> >>
>
> >
>
> >
>
> >>
>
> > > process.stdout.write(received + '\n'); } });
>
> >>
>
> >
>
> >
>
> >>
>
> > > consumer.on('error', function (err) { console.log('error', err); });
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > Not using Mixmax yet?
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac200@gmail.com
>
> >>
>
> >
>
> >
>
> >>
>
> > > wrote:
>
> >>
>
> >
>
> >
>
> >>
>
> > > It sounds like a network issue. Where are the 3 servers located /
> hosted?
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > On Thu, Sep 15, 2016 at 11:51 AM, kant kodali <kanth909@gmail.com>
>
> >>
>
> >
>
> > wrote:
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > Hi,
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > I have the following setup.
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > Single Kafka broker and Zookeeper on Machine 1single Kafka producer on
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > Machine 2
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > Single Kafka Consumer on Machine 3
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > When a producer client sends a message to the Kafka broker by pointing
> at
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > the
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > Zookeeper Server the consumer doesn't seem to get the message right
> away
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > instead
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > it gets after a minute or something (pretty late). I am not sure what
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > settings I
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > need to change. any ideas?
>
> >>
>
> >
>
> >
>
> >>
>
> > >>
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > > Thanks,kant
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>
> >>
>
> > >
>
> >>
>
> >
>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message