kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Luu <manga...@gmail.com>
Subject Suggestions on load testing a system that uses kafka as underlying message bus?
Date Sun, 20 Sep 2015 05:14:32 GMT
I'd like to generate load against a system we have that uses kafka as the
message bus. We have a custom JSON message format, and to properly load
test the system, each set of messages for a particular scenario (i.e. user)
needs to have a unique identifier, which it normally does.

I think of using record & playback technique to capture messages that
correspond to a few users. Then play back those messages to generate load
but to be realistic simulation and to scale up the load, I would:

* re-use the captured user set to simulate additional users to scale up #
of users against the system

* for original set of users and when scaling beyond that for more users, I
would dynamically replace the identifier in the captured messages with a
unique one generated at runtime for each user. also replacing anything else
that needs to like timestamps.

As such, this would have to be a scripted solution. I don't think there is
existing kafka-centric tool to assist with such testing is there?

If not, I'd likely have to build my own. In which case, my question is what
technology stack to use would be most suitable so that I can generate the
highest amount of load with the least amount of load generating
machines/hardware. Using threads, processes, or whatever. node.js, python,
scala, ruby, java, .net, etc.

thoughts, suggestions appreciated,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message