kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Luu <manga...@gmail.com>
Subject Re: Suggestions on load testing a system that uses kafka as underlying message bus?
Date Sun, 20 Sep 2015 18:01:18 GMT
Thanks, Otis, I'll review the info there.

Also, after posting the original message, I came across some kafka
extensions to popular industry (OSS) load test tools:

https://github.com/BrightTag/kafkameter

https://github.com/mnogu/gatling-kafka

they might be useful for tests that require customizations (rather than
standard benchmark tests).

On Sun, Sep 20, 2015 at 7:02 AM, Otis Gospodnetić <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> A few pointers are in this Kafka user ML thread:
> http://search-hadoop.com/m/uyzND177HP92xnm4e
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Sun, Sep 20, 2015 at 1:14 AM, David Luu <mangaroo@gmail.com> wrote:
>
> > I'd like to generate load against a system we have that uses kafka as the
> > message bus. We have a custom JSON message format, and to properly load
> > test the system, each set of messages for a particular scenario (i.e.
> user)
> > needs to have a unique identifier, which it normally does.
> >
> > I think of using record & playback technique to capture messages that
> > correspond to a few users. Then play back those messages to generate load
> > but to be realistic simulation and to scale up the load, I would:
> >
> > * re-use the captured user set to simulate additional users to scale up #
> > of users against the system
> >
> > * for original set of users and when scaling beyond that for more users,
> I
> > would dynamically replace the identifier in the captured messages with a
> > unique one generated at runtime for each user. also replacing anything
> else
> > that needs to like timestamps.
> >
> > As such, this would have to be a scripted solution. I don't think there
> is
> > existing kafka-centric tool to assist with such testing is there?
> >
> > If not, I'd likely have to build my own. In which case, my question is
> what
> > technology stack to use would be most suitable so that I can generate the
> > highest amount of load with the least amount of load generating
> > machines/hardware. Using threads, processes, or whatever. node.js,
> python,
> > scala, ruby, java, .net, etc.
> >
> > thoughts, suggestions appreciated,
> > David
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message