lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Sequence IDs for NRT deletes
Date Tue, 20 Jul 2010 17:03:22 GMT
Breaking up RT patches into baby steps would be great :)  Actually is
the RT branch active (I haven't seen commits going in).

Eg, is per-segment DocWriter separable from the RT changes (seems like
it should/could be)?

The biggest downside of sequence IDs is increase RAM usage right?  Ie,
today each deletion takes 1 bit, but with sequence IDs it's 32X bigger
(an int), I think?  Are there other downsides?

Then, checking if a doc is deleted becomes an int compare instead of a
bit lookup, right?  And, we don't have to clone the deletions during
reopen.

So this is an appropriate tradeoff for apps that need to reopen after
every change to the index.  But for apps reopening less often (eg
maybe up to 10X per second), this may not be a good tradeoff (ie they
are willing to spend more time in the reopen if it reduces RAM
footprint).  Maybe the deletes impl should be pluggable and apps can
pick...

Mike

On Tue, Jul 20, 2010 at 12:33 PM, Jason Rutherglen
<jason.rutherglen@gmail.com> wrote:
> Michael B and I have been discussing the per segment doc writers
> and RT patches/branch. A small improvement we can add to trunk
> from this is the sequence IDs for deletes, which would improve
> the existing NRT system by avoiding the cloning of bit vectors.
> Implementing segment deleted docs via sequence IDs would
> additionally provide a path way for the future RT branch merge
> into trunk. It could be best to break up the RT patches as much
> as possible as they touch on many parts of the Lucene
> IndexWriter subsystem.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message