kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismael Juma <ism...@juma.me.uk>
Subject Re: Broker cannot start switch to Java9 - weird file system issue ?
Date Fri, 20 Apr 2018 18:24:14 GMT
Hi Enrico,

Coincidentally, I saw your message to nio-dev and followed up there.

Ismael

On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli <eolivelli@gmail.com>
wrote:

> It is a deliberate change in JDK code
>
> Just for reference see this discussion  on nio-dev list on OpenJDK
> http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html
>
>
> see
> https://bugs.openjdk.java.net/browse/JDK-8168628
>
> Cheers
> Enrico
>
>
>
> 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <eolivelli@gmail.com>:
>
> > Workaround:
> > as these brokers are only for test environments I have set very small
> > values for index file size, which affects pre-allocation
> > segment.index.bytes=65536
> > log.index.size.max.bytes=65536
> >
> > If anyone has some thought it will be very appreciated
> > Cheers
> >
> > Enrico
> >
> >
> > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <eolivelli@gmail.com>:
> >
> >> The only fact I have found is that with Java8 Kafka is creating "SPARSE"
> >> files and with Java9 this is not true anymore
> >>
> >> Enrico
> >>
> >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <eolivelli@gmail.com>:
> >>
> >>> Hi,
> >>> This is a very strage case. I have a Kafka broker (part of a cluster of
> >>> 3 brokers) which cannot start upgrading Java from Oracle JDK8 to
> Oracle JDK
> >>> 9.0.4.
> >>>
> >>> There are a lot of .index and .timeindex files taking 10MB, they are
> for
> >>> empty partiions.
> >>>
> >>> Running with Java 9 the server seems to rebuild these files and each
> >>> file takes "really" 10MB.
> >>> The sum of all the files (calculated using du -sh) is 22GB and the
> >>> broker crashes during startup, disk becomes full and no log more is
> >>> written. (I can send an extraction of the logs, but the tell only
> about
> >>> 'rebuilding index', the same as on Java 8)
> >>>
> >>> Reverting the same broker to Java 8 and removing the index files, the
> >>> broker rebuilds such files, each files take 10MB, but the full sum of
> sizes
> >>> (calculated using du -sh) is 38 MB !
> >>>
> >>> I am running this broker on CentosOS 7 on EXT4 FS.
> >>>
> >>> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from
> >>> 0.10.2) without any success.
> >>>
> >>> All of the other testing clusters on CentOS7 (same SO settings) did not
> >>> have any problem.
> >>>
> >>> The broker is given 4GB or RAM and 4G of Max Direct Memory Size (on
> >>> Java8 it works fine with 1GB and default max direct memory size)
> >>>
> >>> Which are the relevant configuration options ?
> >>>
> >>> Thoughts ?
> >>>
> >>> Thank you
> >>>
> >>> Enrico
> >>>
> >>>
> >>>
> >>>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message