lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] Commented: (LUCENE-2975) hotspot bug in readvint gives wrong results
Date Sat, 19 Mar 2011 12:38:29 GMT


Uwe Schindler commented on LUCENE-2975:

Before I commit this stuff, I wanted to conclude all Robert and me found out yesterday (in
fact, Robert was doing the debugging work). I just had the idea how to solve it.

The issue has not really something to do with MMap and also is in the Hotspot VM even before
1.6.0_22 (I hit the bug in our server logs for a IndexReader.document() call and were not
able to reproduce, this was 1.6.0_21). So this bug also affects previous Lucene versions,

The bug only happens, if the IndexInput.readByte() method is inlined by hotspot and we are
*not* using BufferedIndexInput (which has its own VInt impl). Lucene 3.0 and prev had a very
complicated readByte() method in MMap, hotspot never inlined. But since an performance update
in MMapIndexInput/MultiMMapIndexInput, the readByte method got a three liner simply delegating
to the ByteBuffer's getByte() and catching an exception to fallback to another impl (for changing
the buffer slice in MultiMMap). So most calls are simply calls to NIOs getByte() which may
be intrinsics or whatever (we don't know what Hotspot does there). Sun optimizes a lot at

This leads to a problem with the loop inside readVInt; all other methods in IndexInput have
already unwinded loops, so readLong is not a "for (i=0; i<64; i+=8)" loop, it is coded
with all shifts precalculated. To solve the bug, I did the same for readVInt and readVLong.

We don't know which bug in hotspot is the real cause for this, so David Weiss's bug looks
really identical, especially as we had in one of our tests also an endless loop (Robert have
seen it, not confirmed). There are a lot of hotspot bugs related to loops, so one of them
hit us here.

> hotspot bug in readvint gives wrong results
> -------------------------------------------
>                 Key: LUCENE-2975
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 3.1
>            Reporter: Uwe Schindler
>            Priority: Blocker
>             Fix For: 3.1
>         Attachments: LUCENE-2975.patch, LUCENE-2975.patch, LUCENE-2975.patch, LUCENE-2975.patch,
> When testing the 3.1-RC1 made by Yonik on the PANGAEA ( productive system
I figured out that suddenly on a large segment (about 5 GiB) some stored fiels suddenly produce
a strange deflate decompression problem (CompressionTools) although the stored fields are
no longer pre-3.0 compressed. It seems that the header of the stored field is read incorrectly
at the buffer boundary in MultiMMapDir and then FieldsReader just incorrectly detects a deflate-compressed
field (CompressionTools).
> The error occurs reproducible on CheckIndex with MMapDirectory, but not with NIODir or
SimpleDir. The FDT file of that segment is 2.6 GiB, on Solaris the chunk size is Integer.MAX_VALUE,
so we have 2 MultiMMap IndexInputs.
> Robert and me have the index ready as a tar file, we will do tests on our local machines
and hopefully solve the bug, maybe introduced by Robert's recent changes to MMap.

This message is automatically generated by JIRA.
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message