kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karolis Pocius <karolis.poc...@adform.com>
Subject Re: log.dirs balance?
Date Tue, 29 Nov 2016 21:04:58 GMT
It's difficult enough to balance kafka brokers with a single log 
directory, not to mention attempting to juggle multiple ones. While JBOD 
is great in terms of capacity, it's a pain in terms of management. After 
6 months of constant manual reassignments I ended up going with RAID1+0 
which is what LinkedIn uses as well as Confluent recommends.

Hats off to you if you manage to find a solution to this, just wanted to 
share my painful experience.


On 2016.11.29 21:35, Tim Visher wrote:
> Hello,
>
> My kafka deploy has 5 servers with 3 log disks each. Over the weekend I
> noticed that on 2 of the 5 servers the partitions appear to be imbalanced
> amongst the log.dirs.
>
> ```
> kafka3
> /var/lib/kafka/disk1
> 3
> /var/lib/kafka/disk2
> 3
> /var/lib/kafka/disk3
> 3
> kafka5
> /var/lib/kafka/disk1
> 3
> /var/lib/kafka/disk2
> 4
> /var/lib/kafka/disk3
> 2
> kafka1
> /var/lib/kafka/disk1
> 3
> /var/lib/kafka/disk2
> 3
> /var/lib/kafka/disk3
> 3
> kafka4
> /var/lib/kafka/disk1
> 4
> /var/lib/kafka/disk2
> 2
> /var/lib/kafka/disk3
> 3
> kafka2
> /var/lib/kafka/disk1
> 3
> /var/lib/kafka/disk2
> 3
> /var/lib/kafka/disk3
> 3
> ```
>
> You can see that 5 and 4 are both unbalanced.
>
> Is there a reason for that? The partitions themselves are pretty much
> perfectly balanced, but the directory chosen for them is not.
>
> Is this an anti-pattern to be using multiple log.dirs per server?
>
> Thanks in advance!
>
> --
>
> In Christ,
>
> Timmy V.
>
> http://blog.twonegatives.com/
> http://five.sentenc.es/ -- Spend less time on mail
>


Best Regards

Karolis Pocius
IT System Engineer

Email: Karolis.Pocius@adform.com
Mobile: +370 620 22108
Sporto g. 18, LT-09238 Vilnius, Lithuania

Disclaimer: 
The information contained in this message and attachments is intended 
solely for the attention and use of the named addressee and may be 
confidential. If you are not the intended recipient, you are reminded that 
the information remains the property of the sender. You must not use, 
disclose, distribute, copy, print or rely on this e-mail. If you have 
received this message in error, please contact the sender immediately and 
irrevocably delete this message and any copies.
Mime
View raw message