lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter
Date Fri, 03 May 2019 08:05:17 GMT
Note that  the -Xint flag will make your code run tremendously more
slowly!  Likely to the point of not really being usable.  But it'd be
interesting to see if that side-steps the bug.

Is it possible to test with OpenJDK as well?

The BKDWriter code is quite complex, so it is also possible there is a
Lucene bug at work.

Can you open an issue in Lucene's jira and we can iterate there?

Thanks,

Mike McCandless

http://blog.mikemccandless.com


On Wed, May 1, 2019 at 9:34 AM Torben Riis <tri@multi-support.com> wrote:

> Hi,
>
>
>
> I’m a bit stuck here and needs a clue or two in order to continue our
> investigations. Hope that someone can help. :)
>
>
>
> Periodically, around once a month, we get the below
> ArrayIndexOutOfBoundsException on our system. We use multiple indexes and
> the error can originate from any of them, but the error always occurs in
> line 1217 in BKDWriter (during a System.arraycopy).
>
>
>
> We found a couple of issues on the net regarding JIT optimization problem
> related to J9, but they all looks like they have been resolved and cannot
> be reproduced anymore. But nevertheless, we have just added the -Xint flag
> (disable JIT compiler) in order to see whether this has any impact.
> Unfortunately we do not have the result of this yet, but I’ll of course
> post it when it is known.
>
>
>
> Are there any of you clever guys out there, that has some good ideas
> further investigations? Or have seen such issue before?
>
>
>
> We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform.
>
>
>
>
>
> *Java version*
>
> java version "1.8.0_191"
>
> Java(TM) SE Runtime Environment (build 8.0.5.25 -
> pap6480sr5fp25-20181030_01(SR5 FP25))
>
> IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References
> 20181029_400846 (JIT enabled, AOT enabled)
>
> OpenJ9   - c5c78da
>
> OMR      - 3d5ac33
>
> IBM      - 8c1bdc2)
>
> JCL - 20181022_01 based on Oracle jdk8u191-b26
>
> NOTICE: If no version information is found above, this could indicate a
> corrupted Java installation!
>
> Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java
>
> -Dmultiarchive.basepath=/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2048m
>
>
>
>
>
> *Stacktrace*
>
> Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST
> [Lucene Merge Thread #0]
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.ArrayIndexOutOfBoundsException
>
>             at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
>
>             at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
>
> Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0]
> java.lang.ArrayIndexOutOfBoundsException
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
>
>             at
> org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
>
>             at
> org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
>
>             at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
>
>             at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
>
>             at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
>
>             at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
>
>             at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
>
> Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:08.075 CEST
> [Lucene Merge Thread #0]
> org.apache.lucene.index.MergePolicy$MergeException:
> java.lang.ArrayIndexOutOfBoundsException
>
>             at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
>
>             at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
>
> Caused by: 2019-05-01T06:10:08.076 CEST [Lucene Merge Thread #0]
> java.lang.ArrayIndexOutOfBoundsException
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
>
>             at
> org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
>
>             at
> org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
>
>             at
> org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
>
>             at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
>
>             at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
>
>             at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
>
>             at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
>
>             at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler
>
>
>
>
>
> *BKDWriter.java*
>
>
>
>
>
>
> Kind regards
>
> Torben Riis
>
> —————————————————————————
> Torben Riis
> Lead Architect
>
>
> Multi Support · Making good business run better
> +45 96 600 600 · www.multi-support.com
>
>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message