kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetić <otis.gospodne...@gmail.com>
Subject Re: Suggestions on load testing a system that uses kafka as underlying message bus?
Date Sun, 20 Sep 2015 14:02:53 GMT
Hi,

A few pointers are in this Kafka user ML thread:
http://search-hadoop.com/m/uyzND177HP92xnm4e

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Sun, Sep 20, 2015 at 1:14 AM, David Luu <mangaroo@gmail.com> wrote:

> I'd like to generate load against a system we have that uses kafka as the
> message bus. We have a custom JSON message format, and to properly load
> test the system, each set of messages for a particular scenario (i.e. user)
> needs to have a unique identifier, which it normally does.
>
> I think of using record & playback technique to capture messages that
> correspond to a few users. Then play back those messages to generate load
> but to be realistic simulation and to scale up the load, I would:
>
> * re-use the captured user set to simulate additional users to scale up #
> of users against the system
>
> * for original set of users and when scaling beyond that for more users, I
> would dynamically replace the identifier in the captured messages with a
> unique one generated at runtime for each user. also replacing anything else
> that needs to like timestamps.
>
> As such, this would have to be a scripted solution. I don't think there is
> existing kafka-centric tool to assist with such testing is there?
>
> If not, I'd likely have to build my own. In which case, my question is what
> technology stack to use would be most suitable so that I can generate the
> highest amount of load with the least amount of load generating
> machines/hardware. Using threads, processes, or whatever. node.js, python,
> scala, ruby, java, .net, etc.
>
> thoughts, suggestions appreciated,
> David
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message