kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Nazemian <alinazem...@gmail.com>
Subject Kafka disk recommendation for cloud
Date Wed, 04 Jul 2018 13:57:37 GMT
Hi All,

I was wondering what the recommendations are for disk type for hosting
Kafka on a cloud environment? As far as I know, most of the best practices
suggest using spinning disks for Kafka due to the fact that Kafka
architecture relies on sequential write/read. Hence, the increase in Kafka
performance by using SSD disks wouldn't be very cost-effective. However, on
a cloud environment, it might be a different story due to hard limitations
on IOPS. For an on-prem solution, the avg IOPS a spinning disk is very low
(about 100-200), but when it comes to sequential IOPS can increase to
20k-30k based on different factors. However, for a cloud solution, there is
a different story. For example, Azure limits every Spinning disk to 500
IOPS whether it is random or sequential. It means we should be able to get
500eps max per each disk, right? Therefore, does it mean using SSD disks
for Kafka would be recommended for cloud providers?

What about using RAID0 vs JBOD for Kafka Brokers? I can see various
recommendations to use RAID0 or JBOD, but I am not really sure which one is
recommended especially for a Cloud environment?

Regards,
Ali

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message