kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kurt Rudolph <rudolph.k...@gmail.com>
Subject Re: Kafka hardware requirements
Date Mon, 06 May 2019 13:22:24 GMT
I am currently running a few clusters on virtual machines connected to a
VSAN.  I have not had any major ssues.  My clusters are small; 3 virtual
machines  (4 cpu and 8 GB memory each).  I am routinely processing 20K
msg/sec with surges of over 35K msg/sec.  Based on my performance testing,
I don't think my cluster would have issues until I start reaching about 80K
msg/sec.  Your message throughput will be highly dependent on your message
size and whether you are compressing messages.  You can test our the
performance of your cluster using kafka-producer-perf-test and
kafka-consumer-perf-test.  This should give you a good idea of what your
configuration  can do.

The only issue I've had running on vms or a vsan is when VMWare itself has
issues.  Our admin had to reshuffle the VSAN one day and that caused high
IOWait on my cluster and caused a number of partitions to become out
of sync with the leader.

On Sun, May 5, 2019 at 2:50 PM Jean-Marc Hyppolite <
jean.marc.hyppolite@outlook.com> wrote:

> Hello,
>
> I would like to know the impact of running kafka in production on virtual
> machines connected to a SAN (Storage Area Network)? (I mean the impact on
> kafka performance, what would be a max limit in terms of number of messages
> per second, number of producers and consumers, kafka can deal with ...). If
> we have no choice of running kafka on that kind of "hardware" what would be
> the deployment guidelines along the with the max limit of number of
> messages per second, of producers and consumers?
>
> Thank you.
>
> Jean-Marc.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message