kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Olivelli <eolive...@gmail.com>
Subject Broker cannot start switch to Java9 - weird file system issue ?
Date Mon, 05 Mar 2018 11:44:03 GMT
Hi,
This is a very strage case. I have a Kafka broker (part of a cluster of 3
brokers) which cannot start upgrading Java from Oracle JDK8 to Oracle JDK
9.0.4.

There are a lot of .index and .timeindex files taking 10MB, they are for
empty partiions.

Running with Java 9 the server seems to rebuild these files and each file
takes "really" 10MB.
The sum of all the files (calculated using du -sh) is 22GB and the broker
crashes during startup, disk becomes full and no log more is written. (I
can send an extraction of the logs, but the tell only  about 'rebuilding
index', the same as on Java 8)

Reverting the same broker to Java 8 and removing the index files, the
broker rebuilds such files, each files take 10MB, but the full sum of sizes
(calculated using du -sh) is 38 MB !

I am running this broker on CentosOS 7 on EXT4 FS.

I have upgraded the broker to latest and greatest Kafka 1.0.0 (from 0.10.2)
without any success.

All of the other testing clusters on CentOS7 (same SO settings) did not
have any problem.

The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on Java8
it works fine with 1GB and default max direct memory size)

Which are the relevant configuration options ?

Thoughts ?

Thank you

Enrico

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message