kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Olivelli <eolive...@gmail.com>
Subject Re: Broker cannot start switch to Java9 - weird file system issue ?
Date Mon, 05 Mar 2018 12:21:10 GMT
The only fact I have found is that with Java8 Kafka is creating "SPARSE"
files and with Java9 this is not true anymore

Enrico

2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eolivelli@gmail.com>:

> Hi,
> This is a very strage case. I have a Kafka broker (part of a cluster of 3
> brokers) which cannot start upgrading Java from Oracle JDK8 to Oracle JDK
> 9.0.4.
>
> There are a lot of .index and .timeindex files taking 10MB, they are for
> empty partiions.
>
> Running with Java 9 the server seems to rebuild these files and each file
> takes "really" 10MB.
> The sum of all the files (calculated using du -sh) is 22GB and the broker
> crashes during startup, disk becomes full and no log more is written. (I
> can send an extraction of the logs, but the tell only  about 'rebuilding
> index', the same as on Java 8)
>
> Reverting the same broker to Java 8 and removing the index files, the
> broker rebuilds such files, each files take 10MB, but the full sum of sizes
> (calculated using du -sh) is 38 MB !
>
> I am running this broker on CentosOS 7 on EXT4 FS.
>
> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from
> 0.10.2) without any success.
>
> All of the other testing clusters on CentOS7 (same SO settings) did not
> have any problem.
>
> The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on Java8
> it works fine with 1GB and default max direct memory size)
>
> Which are the relevant configuration options ?
>
> Thoughts ?
>
> Thank you
>
> Enrico
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message