kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörg Wagner <joerg.wagn...@1und1.de>
Subject Distribution of logs over log.dirs
Date Tue, 04 Jul 2017 09:24:24 GMT

I have a setup with 3 brokers having 27 log.dirs each. Setting up a 
topic with 27 partitions and 2 replication I would have expected every 
directory of log.dirs to feature <1 partition. This is not the case (the 
setup has been the same from the beginning; e.g. no log.dirs added later).

If you could explain how the distribution is calculated I would be very 
grateful. Also: is it possible to move data from one directory to 
another since it's very unbalanced currently?

Another issue I have is that looking into this I noticed the replicas 
use about half the space of the leading partition, though the 
log.cleaner (and general broker) configuration on all brokers is 
identical. This leads me to think either the leader uses more space (not 
really) or the cleaner is not working on one broker. I compared one of 
the partition directory on different brokers and the amount of files is 
also far more for the aforementioned leader, also pointing to the 
cleaner not working. But why?
I compared configs (which is automated, hence no manual errors possible) 
and am very stumped about this behaviour.


View raw message