lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Babak Farhang <>
Subject Re: Incremental Field Updates
Date Sat, 03 Apr 2010 05:49:56 GMT

Reading your post a 3rd time, I see my "suggestion" is in fact the
approach you describe.  Sorry for being redundant.


On Fri, Apr 2, 2010 at 11:25 PM, Babak Farhang <> wrote:
>> I think they get merged in by the merger, ideally in the background.
> That sounds sensible. (In other words, we wont concern ourselves with
> roll backs--something possible while a "layer" is still around.)
> I've been thinking about this problem also. One approach discussed
> earlier in these mailing lists has been to somehow maintain a parallel
> index of the update-able of the fields in such a way that the docIds
> of the parallel index remain in sync with the "master" index. Mike
> McCandless and I were discussing some variants of this approach a few
> months back:
>  That approach involved the concept of mapping (chaining, if you will)
> internal docIds to view ids.  That docid mapping concept sounds
> analogous to this layer concept we are discussing now.
> I now think the parallel index approach may not be such a great idea,
> after all: it simply pushes the problem to the edge--the slave index.
> If we can solve update problem in the slave index, I reason, then
> shouldn't we also be able to solve the same update problem in the
> master index (and thereby remove the necessity of maintaining a
> (user-level) parallel index in the first place)?
> Which seems to align with the approach being discussed here..
> I imagine the "layers" being discussed here are somehow threaded by
> docId. That is, given a docId, you can quickly find it's "layers."  If
> so, then the docId mapping idea may be one way to thread these layers.
> (A logical document would be constructed by a chain of docIds, each
> overriding the previous for each field it defines (or deletes).  Such
> a construction would have to be "merge-aware" (perhaps using machinery
> similar to that used in LUCENE-1879) in order that it may maintain the
> docId chain.
> What do you think?
> On Fri, Apr 2, 2010 at 4:56 AM, Grant Ingersoll <> wrote:
>> On Apr 2, 2010, at 2:50 AM, Babak Farhang wrote:
>>> [Late to this party, but thought I'd chime in]
>>> I think this "layer" concept is right on.  But I'm wondering about the
>>> life cycle of these layers.  Do layers live forever? Or do they
>>> collapse at some point? (Like, as I think was already pointed out,
>>> deletes are when segments are merged today.)
>> I think they get merged in by the merger, ideally in the background.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message