kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismael Juma <ism...@juma.me.uk>
Subject Re: Broker cannot start switch to Java9 - weird file system issue ?
Date Thu, 26 Apr 2018 18:05:43 GMT
Thanks, I updated it.

On Thu, Apr 26, 2018 at 4:09 AM, Enrico Olivelli <eolivelli@gmail.com>
wrote:

> Here it is !
>
> https://issues.apache.org/jira/browse/KAFKA-6828
>
> Thank you
>
> Enrico
>
> 2018-04-24 20:53 GMT+02:00 Ismael Juma <ismael@juma.me.uk>:
>
> > A JIRA ticket would be appreciated. :)
> >
> > Ismael
> >
> > On Sat, Apr 21, 2018 at 12:51 AM, Enrico Olivelli <eolivelli@gmail.com>
> > wrote:
> >
> > > Il sab 21 apr 2018, 06:29 Ismael Juma <ismael@juma.me.uk> ha scritto:
> > >
> > > > Hi Enrico,
> > > >
> > > > It is a real problem because it causes indexes to take a lot more
> disk
> > > > space upfront. The sparsity is an important if people over partition,
> > for
> > > > example.
> > > >
> > >
> > > Got it.
> > > In production I saw no issue, maybe due to much availability of disk
> > space.
> > >
> > > Should I file a JIRA or you will do?
> > > Enrico
> > >
> > >
> > > > Ismael
> > > >
> > > > On Fri, Apr 20, 2018 at 12:41 PM, Enrico Olivelli <
> eolivelli@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Il ven 20 apr 2018, 20:24 Ismael Juma <ismael@juma.me.uk> ha
> > scritto:
> > > > >
> > > > > > Hi Enrico,
> > > > > >
> > > > > > Coincidentally, I saw your message to nio-dev and followed up
> > there.
> > > > > >
> > > > >
> > > > > I think this is not a 'real' problem, you will notice the
> difference
> > > only
> > > > > if you have a lot of empty topics/partitions.
> > > > > In fact when you simply upgrade to jdk9/10 immediately you are
> > charged
> > > > with
> > > > > 10MB of disk space for each partition.
> > > > > In production I did not suffer this change because usually you do
> not
> > > > have
> > > > > empty partitions.
> > > > > In my test environments, where I had thousands of test empty
> > > partitions,
> > > > > disks filled up immediately and without any reason, it took time
to
> > > > > understand the real cause. Broker will crash without much 'log' as
> > disk
> > > > is
> > > > > out of space.
> > > > >
> > > > > Maybe it would be useful to add some notice about this potential
> > > problem
> > > > > during the upgrade of the jdk.
> > > > >
> > > > > Hope that helps
> > > > > Enrico
> > > > >
> > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Fri, Apr 20, 2018 at 8:18 AM, Enrico Olivelli <
> > > eolivelli@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > It is a deliberate change in JDK code
> > > > > > >
> > > > > > > Just for reference see this discussion  on nio-dev list
on
> > OpenJDK
> > > > > > >
> > > > http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/
> 005008.html
> > > > > > >
> > > > > > >
> > > > > > > see
> > > > > > > https://bugs.openjdk.java.net/browse/JDK-8168628
> > > > > > >
> > > > > > > Cheers
> > > > > > > Enrico
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 2018-03-05 14:29 GMT+01:00 Enrico Olivelli <
> eolivelli@gmail.com
> > >:
> > > > > > >
> > > > > > > > Workaround:
> > > > > > > > as these brokers are only for test environments I
have set
> very
> > > > small
> > > > > > > > values for index file size, which affects pre-allocation
> > > > > > > > segment.index.bytes=65536
> > > > > > > > log.index.size.max.bytes=65536
> > > > > > > >
> > > > > > > > If anyone has some thought it will be very appreciated
> > > > > > > > Cheers
> > > > > > > >
> > > > > > > > Enrico
> > > > > > > >
> > > > > > > >
> > > > > > > > 2018-03-05 13:21 GMT+01:00 Enrico Olivelli <
> > eolivelli@gmail.com
> > > >:
> > > > > > > >
> > > > > > > >> The only fact I have found is that with Java8
Kafka is
> > creating
> > > > > > "SPARSE"
> > > > > > > >> files and with Java9 this is not true anymore
> > > > > > > >>
> > > > > > > >> Enrico
> > > > > > > >>
> > > > > > > >> 2018-03-05 12:44 GMT+01:00 Enrico Olivelli <
> > eolivelli@gmail.com
> > > >:
> > > > > > > >>
> > > > > > > >>> Hi,
> > > > > > > >>> This is a very strage case. I have a Kafka
broker (part of
> a
> > > > > cluster
> > > > > > of
> > > > > > > >>> 3 brokers) which cannot start upgrading Java
from Oracle
> JDK8
> > > to
> > > > > > > Oracle JDK
> > > > > > > >>> 9.0.4.
> > > > > > > >>>
> > > > > > > >>> There are a lot of .index and .timeindex files
taking 10MB,
> > > they
> > > > > are
> > > > > > > for
> > > > > > > >>> empty partiions.
> > > > > > > >>>
> > > > > > > >>> Running with Java 9 the server seems to rebuild
these files
> > and
> > > > > each
> > > > > > > >>> file takes "really" 10MB.
> > > > > > > >>> The sum of all the files (calculated using
du -sh) is 22GB
> > and
> > > > the
> > > > > > > >>> broker crashes during startup, disk becomes
full and no log
> > > more
> > > > is
> > > > > > > >>> written. (I can send an extraction of the
logs, but the
> tell
> > > only
> > > > > > > about
> > > > > > > >>> 'rebuilding index', the same as on Java 8)
> > > > > > > >>>
> > > > > > > >>> Reverting the same broker to Java 8 and removing
the index
> > > files,
> > > > > the
> > > > > > > >>> broker rebuilds such files, each files take
10MB, but the
> > full
> > > > sum
> > > > > of
> > > > > > > sizes
> > > > > > > >>> (calculated using du -sh) is 38 MB !
> > > > > > > >>>
> > > > > > > >>> I am running this broker on CentosOS 7 on
EXT4 FS.
> > > > > > > >>>
> > > > > > > >>> I have upgraded the broker to latest and greatest
Kafka
> 1.0.0
> > > > (from
> > > > > > > >>> 0.10.2) without any success.
> > > > > > > >>>
> > > > > > > >>> All of the other testing clusters on CentOS7
(same SO
> > settings)
> > > > did
> > > > > > not
> > > > > > > >>> have any problem.
> > > > > > > >>>
> > > > > > > >>> The broker is given 4GB or RAM and 4G of Max
Direct Memory
> > Size
> > > > (on
> > > > > > > >>> Java8 it works fine with 1GB and default max
direct memory
> > > size)
> > > > > > > >>>
> > > > > > > >>> Which are the relevant configuration options
?
> > > > > > > >>>
> > > > > > > >>> Thoughts ?
> > > > > > > >>>
> > > > > > > >>> Thank you
> > > > > > > >>>
> > > > > > > >>> Enrico
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > --
> > > > >
> > > > >
> > > > > -- Enrico Olivelli
> > > > >
> > > >
> > > --
> > >
> > >
> > > -- Enrico Olivelli
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message