kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Akhtar <ali.rac...@gmail.com>
Subject Re: hi
Date Thu, 15 Sep 2016 08:54:35 GMT
I'd post to the mailing list again with a new subject and ask that.

On Thu, Sep 15, 2016 at 1:52 PM, kant kodali <kanth909@gmail.com> wrote:

> I used node.js client libraries for all three and yes I want to make sure
> I am
> comparing apples to apples so I make it as equivalent as possible.
> Again the big question is What is the right setup for Kafka to be
> comparable
> with the other I mentioned in my previous email?
>
>
>
>
>
>
> On Thu, Sep 15, 2016 1:47 AM, Ali Akhtar ali.rac200@gmail.com
> wrote:
> The issue is clearly that you're running out of resources, so I would add
>
> more brokers and/or larger instances.
>
>
>
>
> You're also using Node which is not the best for performance. A compiled
>
> language such as Java would give you the best performance.
>
>
>
>
> Here's a case study that should help:
>
> https://engineering.linkedin.com/kafka/benchmarking-apache-k
> afka-2-million-writes-second-three-cheap-machines
>
>
>
>
> Good luck, let us know how it goes
>
>
>
>
> On Thu, Sep 15, 2016 at 1:42 PM, kant kodali <kanth909@gmail.com> wrote:
>
>
>
>
> yeah..
>>
>
> I tried it with 10 messages with single broker and only one partiton that
>>
>
> looked
>>
>
> instantaneous and ~5K messages/sec for the data size of 1KBI tried it with
>>
>
> 1000
>>
>
> messages that looked instantaneous as well ~5K messages/sec for the data
>>
>
> size of
>>
>
> 1KBI tried it with 10K messages with single broker and only one
>>
>
> partiton things
>>
>
> started to go down ~1K messages/sec for the data size of 1KB
>>
>
> having only one partition on a single broker is a bad? My goal is to run
>>
>
> some
>>
>
> basic benchmarks on NATS & NSQ & KAFKA
>>
>
> I have the same environment for all three (NATS & NSQ & KAFKA)
>>
>
> a broker on Machine 1producer on Machine 2Consumer on Machine 3
>>
>
> with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
>>
>
> instance.
>>
>
> I have pushed 300K messages with NATS and it was able to handle easily and
>>
>
> receive throughput was 5K messages/secI have pushed 300K messages and NSQ
>>
>
> and
>>
>
> receive throughput was 2K messages/secI am unable to push 300K messages
>>
>
> with
>>
>
> Kafka with the above configuration and environment so at this point my
>>
>
> biggest
>>
>
> question is what is the fair setup for Kafka so its comparable with NATS
>>
>
> and
>>
>
> NSQ?
>>
>
> kant
>>
>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 12:43 AM, Ali Akhtar ali.rac200@gmail.com
>>
>
> wrote:
>>
>
> Lower the workload gradually, start from 10 messages, increase to 100, then
>>
>
>
>>
> 1000, and so on. See if it slows down as the workload increases. If so, you
>>
>
>
>>
> need more brokers + partitions to handle the workload.
>>
>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 at 12:42 PM, kant kodali <kanth909@gmail.com> wrote:
>>
>
>
>>
>
>>
>
>>
>
>>
> > m4.xlarge
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac200@gmail.com
>>
>
>
>>
> >
>>
>
>
>>
> > wrote:
>>
>
>
>>
> > What's the instance size that you're using? With 300k messages your
>>
>
> single
>>
>
>
>>
> >
>>
>
>
>>
> > broker might not be able to handle it.
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > On Thu, Sep 15, 2016 at 12:30 PM, kant kodali <kanth909@gmail.com>
>>
>
> wrote:
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > My goal is to test the throughput (#messages per second) given my setup
>>
>
> and
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > with a data size of 1KB. if you guys already have some idea on these
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > numbers
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > that would be helpful as well.
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth909@gmail.com
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > wrote:
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > 172.* is all private ip's for my machine I double checked it.I have not
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > changed
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > any default settingsI dont know how to use kafka-consumer.sh
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > or kafka-producer.sh because it looks like they want me to specify a
>>
>
> group
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > and I
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > didn't create any consumer group because I am using single producer and
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > consumer. is there a default group?Also, I am receiving message but very
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > late. I
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > send about 300K messages using the node.js client and I am receiving at a
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > very
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > low rate. really not sure what is going on?
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac200@gmail.com
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > wrote:
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Your code seems to be using the public ip of the servers. If all 3
>>
>
> machines
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > are in the same availability zone on AWS, try using the private ip, and
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > then they might communicate over the local network.
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > Did you change any default settings?
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > Do you get the same results if you run kafka-consumer.sh and
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > kafka-producer.sh instead of the Node code?
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > On Thu, Sep 15, 2016 at 12:01 PM, kant kodali <kanth909@gmail.com>
>>
>
> wrote:
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > They are hosted on AWS and I dont think there are any network issues
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > because I
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > tried testing other Queuing systems with no issues however I am using a
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > node.js
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > client with the following code. I am not sure if there are any errors
>>
>
> or
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > anything I didn't set in the following code?
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > //producer var kafka = require('kafka-node'); var
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > Producer = kafka.Producer; var Client = kafka.Client; var client =
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > new Client('172.31.21.175:2181'); var argv =
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > p = argv.p || 0; var a = argv.a || 0; var producer = new
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > Producer(client, { requireAcks: 1}); var num = 300005;
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > producer.on('ready', function () { var message = 'Hello World';
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > for (var i=0; i<num; i++) { producer.send([ { topic:
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > topic, partition: p, messages: message, attributes: a } ], function
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > (err, result) { console.log(err || result);
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > //process.exit(); }); } }); producer.on('error',
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > function (err) { console.log('error', err); process.exit();
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > }); //Consumer var kafka = require('kafka-node'); var Consumer =
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > kafka.Consumer; var Offset = kafka.Offset; var Client =
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > kafka.Client; var argv = require('optimist').argv; var topic =
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > argv.topic || 'kafka_test'; var client = new
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > Client('172.31.21.175:2181'); var topics = [ {topic: topic,
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > 1000 }; var consumer = new Consumer(client, topics, options); var
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > offset = new Offset(client); var start; var received = 0; var
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > target = 200000; var hash = 1000; consumer.on('message', function
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > (message) { console.log(message); received += 1; if
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > (received === 1) { start = new Date(); } if (received === target) {
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > var stop = new Date(); console.log('\nDone test');
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > var mps = parseInt(target/((stop-start)/1000));
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > console.log('Received at ' + mps + ' msgs/sec'); process.exit();
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > } else if (received % hash === 0){
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > > process.stdout.write(received + '\n'); } });
>>
>
>
>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message