lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <>
Subject Re: [DISCUSS] Opening old indices for reading
Date Tue, 29 Jan 2019 15:23:04 GMT
thanks folks,

these are all good points. I created a first cut of what I had in mind
[1] . It's relatively simple and from a java visibility perspective
the only change that a user can take advantage of is this [2] and this
[3] respectively. This would allow opening indices back to Lucene 7.0
given that the codecs and postings formats are available. From a
documentation perspective I added [4]. Thisi s a pure read-only change
and doesn't allow opening these indices for writing. You can't merge
them neither would you be able to open an index writer on top of it. I
still need to add support to Check-Index but that's what it is

lemme know what you think,


On Fri, Jan 25, 2019 at 3:14 PM Michael McCandless
<> wrote:
> Another example is long ago Lucene allowed pos=-1 to be indexed and it caused all sorts
of problems.  We also stopped allowing positions close to Integer.MAX_VALUE (
 Yet another is allowing negative vInts which are possible but horribly inefficient (
> We do need to be free to fix these problems and then know after N+2 releases that no
index can have the issue.
> I like the idea of providing "expert" / best effort / limited way of carrying forward
such ancient indices, but I think the huge challenge for someone using that tool on an important
index will be enumerating the list of issues that might "matter" (the 3 Adrien listed + the
3 I listed above is a start for this list) and taking appropriate steps to "correct" the index
if so.  E.g. on a norms encoding change, somehow these expert tools must decode norms the
old way, encode them the new way, and then rewrite the norms files.  Or if the index has pos=-1,
changing that to pos=0.  Or if it has negative vInts, ... etc.
> Or maybe the "special" DirectoryReader only reads stored fields?  And so you would enumerate
your _source and reindex into the latest format ...
> > Something like would
> > help make it harder to introduce corrupt data in an index.
> +1
> Every time we catch something like "don't allow pos = -1 into the index" we need somehow
remember to go and add the check also in addIndices.
> Mike McCandless
> On Fri, Jan 25, 2019 at 3:52 AM Adrien Grand <> wrote:
>> Agreed with Michael that setting expectations is going to be
>> important. The thing that I would like to make sure is that we would
>> never refrain from moving Lucene forward because of this feature. In
>> particular, lucene-core should be free to make assumptions that are
>> valid for N and N-1 indices without worrying about the fact that we
>> have this super-expert feature that allows opening older indices. Here
>> are some assumptions that I have in mind which have not always been
>> true:
>>  - norms might be encoded in a different way (this changed in 7)
>>  - all index files have a checksum (only true since Lucene 5)
>>  - offsets are always going forward (only enforced since Lucene 7)
>> This means that carrying indices over by just merging them with the
>> new version to move them to a new codec won't work all the time. For
>> instance if your index has backward offsets and new codecs assume that
>> offsets are going forward, then merging might fail or corrupt offsets
>> - I'd like to make sure that we would not consider this a bug.
>> Erick, I don't think this feature would be suitable for "robust index
>> upgrades". To me it is really a best effort and shouldn't be trusted
>> too much.
>> I think some users will be tempted to wrap old readers to make them
>> look good and then add them back to an index using addIndexes?
>> Something like would
>> help make it harder to introduce corrupt data in an index.
>> On Wed, Jan 23, 2019 at 3:11 PM Simon Willnauer
>> <> wrote:
>> >
>> > Hey folks,
>> >
>> > tl;dr; I want to be able to open an indexreader on an old index if the
>> > SegmentInfo version is supported and all segment codecs are available.
>> > Today that's not possible even if I port old formats to current
>> > versions.
>> >
>> > Our BWC policy for quite a while has been N-1 major versions. That's
>> > good and I think we should keep it that way. Only recently, caused by
>> > changes how we encode/decode norms we also hard-enforce a the
>> > index-version-created in several places and the version a segment was
>> > written with. These are great enforcements and I understand why. My
>> > request here is if we can find consensus on allowing somehow (a
>> > special DirectoryReader for instance) to open such an index for
>> > reading only that doesn't provide the guarantees that our high level
>> > APIs decode norms correctly for instance. This would be enough to for
>> > instance consume stored fields etc. for reindexing or if a users are
>> > aware do they norms decoding in the codec. I am happy to work on a
>> > proposal how this would work. It would still enforce no writing or
>> > anything like this. I am also all for putting such a reader into misc
>> > and being experimental.
>> >
>> > simon
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail:
>> > For additional commands, e-mail:
>> >
>> --
>> Adrien
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message