lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Babak Farhang <>
Subject Re: Incremental Field Updates
Date Fri, 09 Apr 2010 02:05:45 GMT
Good point. I meant the model at the document level: i.e. what
milestones does a document go through in its life cycle. Today:

created --> deleted

With incremental updates:

created --> update1 --> update2 --> deleted

I think what I'm trying to say is that this second threaded sequence
of state changes seems intuitively more fragile under concurrent
scenarios.  So for example, in a lock-free design, the system would
also have to anticipate the following sequence of events:

created --> update1 --> deleted --> update2

and consider update2 a null op.  I'm imagining there are other cases
that I can't think of..


On Tue, Apr 6, 2010 at 3:40 AM, Michael McCandless
<> wrote:
> write once, plus the option to the app to keep multiple commit points
> around (by customizing the deletion policy).
> Actually order of operations / commits very much matters in Lucene today.
> Deletions are not idempotent: if you add a doc w/ term X, delete by
> term X, add a new doc with term X... that's very different than if you
> moved the delete op to the end.  Ie the deletion only applies to the
> docs added before it.
> Mike
> On Mon, Apr 5, 2010 at 12:45 AM, Babak Farhang <> wrote:
>> Sure. Because of the write once principle.  But at some cost
>> (duplicated data). I was just agreeing that it would not be a good
>> idea to bake in version-ing by keeping the layers around forever in a
>> merged index; I wasn't keying in on transactions per se.
>> Speaking of transactions: I'm not sure if we should worry about this
>> much yet, but with "updates" the order of the transaction commits
>> seems important. I think commit order is less important today in
>> Lucene because its model supports only 2 types of events: document
>> creation--which only happens once, and document deletion, which is
>> idempotent.  What do you think? Will commits have to be ordered if we
>> introduce updates?  Or does the onus of maintaining order fall on the
>> application?
>> -Babak
>> On Sat, Apr 3, 2010 at 3:28 AM, Michael McCandless
>> <> wrote:
>>> On Sat, Apr 3, 2010 at 1:25 AM, Babak Farhang <> wrote:
>>>>> I think they get merged in by the merger, ideally in the background.
>>>> That sounds sensible. (In other words, we wont concern ourselves with
>>>> roll backs--something possible while a "layer" is still around.)
>>> Actually roll backs would still be very possible even if layers are merged.
>>> Ie, one could keep multiple commits around, and the older commits
>>> would still be referring to the old postings + layers, keeping them
>>> alive.
>>> Lucene would still be transactional with such an approach.
>>> Mike
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> For additional commands, e-mail:
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message