lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <>
Subject Re: Back Compatibility
Date Wed, 23 Jan 2008 16:54:31 GMT
Maybe I don't understand lockless commits then.

I just don't think you can enforce transactional consistency without  
either 1) locking, or 2) optimistic collision detection. I could be  
wrong here, but this has been my experience.

By effectively removing the locking requirement, I think you are  
going to have users developing code without thought as to what is  
going to happen when locking is added. This is going to break the  
backwards compatibility that people are striving for.

The lucene "writer" structure needs to be something like:

start tx for update
do work

where commit is composed of (prepare and commit phases), but commit  
may fail.

It is unknown if this can actually happen though, since there is no  
unique ID that could cause collisions, but there is the internal id  
(which would need to remain constant throughout the tx in order for  
queries and delete operations to work).

I am sure it is that I don't understand lockless commits, so I will  
give a scenario.

client A issues query looking for documents with OID (a field) =  
"some field";
client B issues same query
both queries return nothing found
client A inserts document with OID = "some filed"
client B inserts document with OID = "some field"

client A commits and client B commits

unless B is blocked, once A issues the query, the index is going to  
end up with 2 different copies of the document.

I understand that Lucene is not a database, and has no concept of  
unique constraints. It is my understand that this has been overcome  
using locks and sequential access to the index when writing.

In a simple XA implementation, client A would open a SERIALIZABLE  
transaction, which would block B from even reading the index.  Most  
simple XA implementation only support READ_COMMITTED, SERIALIZABLE,  
and NONE.

There are other ways of offering finer grained locking (based on  
internal id and timestamps), but most are going to need a "server  
based" implementation of lucene to pull off.

To summarize, I think the "shared filestore (NFS)" and "lockless  
commits" make implementing transactions very difficult. I am sure I  
am missing something here, I just don't see what.

On Jan 23, 2008, at 8:53 AM, Mark Miller wrote:

> Thats where Robert is confusing me as well. To have XA support you  
> just need to be able to define a transaction, atomically commit, or  
> rollback. You also need a consistent state after any of these  
> operations. LUCENE-1044 seems to guarantee that, and so isn't it  
> more like finishing up needed work than going down the wrong path?  
> It seems more to me (and obviously I know a lot less about this  
> than either of you) that you have just gotten Lucene ready to add  
> XA support. Lucene now fulfills all of the requirements. No?  
> Someone just needs to write a boatload of JTA code :)
> It would seem the next step would be, as Robert suggests, to make a  
> transaction a first class citizen. The XA protocol will require  
> Lucene to communicate with the TM about what transactions it has  
> completed to help in failure recovery and transaction management. I  
> can certainly see the need for a better transaction abstraction to  
> help with this.
> A little enlightenment on this would be great robert. I am very  
> interested in it for future projects.
> And I have to point just seems logical that we would make  
> things so that the index was consistent at some point before taking  
> the next step of making it consistent with other I  
> am just still confused about Roberts objections to what is going on  
> here. I think that it would be a real leap forward to get it done  
> though.
> Also, as he mentioned, we really need a good distributed system  
> that allows for index partitioning. Thats the ticket to more  
> enterprise adoption. Could be Solr's work though...
> Michael McCandless wrote:
>> Robert, besides LUCENE-1044 (syncing on commit), what is the Lucene
>> core missing in order for you (or, someone) to build XA compliance on
>> top of it?
>> Ie, you can open a writer with autoCommit=false and no changes are
>> committed until you close it.  You can abort the session by calling
>> writer.abort().  What's still missing, besides LUCENE-1044?
>> Mike
>> robert engels wrote:
>>> One more example on this. A lot of work was done on transaction  
>>> support. I would argue that this falls way short of what is  
>>> needed, since there is no XA transaction support. Since the  
>>> lucene index (unless stored in an XA db) is a separate resource,  
>>> it really needs XA support in order to be consistent with the  
>>> other resources.
>>> All of the transaction work that has been performed only  
>>> guarantees that barring a physical hardware failure the lucene  
>>> index can be opened and used at a known state.  This index though  
>>> is probably not consistent with the other resources.
>>> All that was done is that we can now guarantee that the index is  
>>> consistent at SOME point in time.
>>> Given the work that was done, we are probably closer to adding XA  
>>> support, but I think this would be much easier if the concept of  
>>> a transaction was made first class through the API (and then XA  
>>> transactions need to be supported).
>>> On Jan 22, 2008, at 2:49 PM, robert engels wrote:
>>>> I don't think group C is interested in bug fixes. I just don't  
>>>> see how Lucene is at all useful if the users are encountering  
>>>> any bug - so they either don't use that feature, or they have  
>>>> already developed a work-around (or they have patched the code  
>>>> in a way that avoids the bug, yet is specific to their  
>>>> environment).
>>>> For example, I think the NFS work (bugs, fixes, etc.) was quite  
>>>> substantial. I think the actual number of people trying to use  
>>>> NFS is probably very low - as the initial implementation had so  
>>>> many problems (and IMO is not a very good solution for  
>>>> distributed indexes anyway). So all the work in trying to make  
>>>> NFS work "correctly" behind the scenes may have been  
>>>> inefficient, since a more direct, yet major fix may have solved  
>>>> the problem better (like distributed server support, not shared  
>>>> index access).
>>>> I just think that trying to maintain API compatibility through  
>>>> major releases is a bad idea. Leads to bloat, and complex code -  
>>>> both internal and external.  In order to achieve great gains in  
>>>> usability and/or performance in a mature product like Lucene  
>>>> almost certainly requires massive changes to the processes,  
>>>> algorithms and structures, and the API should change as well to  
>>>> reflect this.
>>>> On Jan 22, 2008, at 2:30 PM, Chris Hostetter wrote:
>>>>> : If they are " no longer actively developing the portion of  
>>>>> the code that's
>>>>> : broken, aren't seeking the new feature, etc", and they stay  
>>>>> back on old
>>>>> : versions... isn't that exactly what we want? They can stay on  
>>>>> the old version,
>>>>> : and new application development uses the newer version.
>>>>> This basically mirrors a philosophy that is rising in the Perl
>>>>> community evangelized by (a really smart dude named chromatic) ...
>>>>> "why are we worry about the effect of upgrades on users who  
>>>>> don't upgrade?"
>>>>> The problem is not all users are created equal and not all  
>>>>> users upgrade
>>>>> for the same reasons or at the same time...
>>>>> Group A: If someone is paranoid about upgrading, and is still  
>>>>> running
>>>>> lucene1.4.3 because they are afraid if they upgrade their app  
>>>>> will break
>>>>> and they don't want to deal with it; they don't care about  
>>>>> known bugs in
>>>>> lucene1.4.3, as long as those bugs haven't impacted them yet --  
>>>>> these
>>>>> people aren't going to care wether we add a bunch of new  
>>>>> methods to
>>>>> interfaces, or remove a bunch of public methods from arbitrary  
>>>>> releases,
>>>>> because they are never going to see them.  They might do a  
>>>>> total rewrite
>>>>> of their project later, and they'll worry about it then (when  
>>>>> they have
>>>>> lots of time and QA resources)
>>>>> Group: B: At the other extreme, are the "free-spirited"  
>>>>> developers (god i
>>>>> hate that that the word "agile" has been co-opted) who are  
>>>>> always eager to
>>>>> upgrade to get the latest bells and whistles, and don't mind  
>>>>> making
>>>>> changes to code and recompiling everytime they upgrades -- just  
>>>>> as long as
>>>>> there are some decent docs on what to change.
>>>>> Croup: C: In the middle is a larg group of people who are  
>>>>> interested in
>>>>> upgrading, who want bug fixes, are willing to write new code to  
>>>>> take
>>>>> advantage of new features, in some cases are even willing to make
>>>>> small or medium changes their code to get really good performance
>>>>> improvements ... but they don't have a lot of time or energy to  
>>>>> constantly
>>>>> rewrite big chunks of their app.  For these people, knowing  
>>>>> that they can
>>>>> "drop in" the new version and it will work is a big reason why  
>>>>> there are
>>>>> willing to upgrade, and why they are willing to spend soem time
>>>>> tweaking code to take advantage of the new features and the new
>>>>> performacne enhaced APIs -- becuase they don't have to spend a  
>>>>> lot of time
>>>>> just to get the app working as well as it was before.
>>>>> To draw an analogy...
>>>>> Group A will stand in one place for a really long time no  
>>>>> matter how easy
>>>>> the path is.  Once in a great while they will decide to march  
>>>>> forward
>>>>> dozens of miles in one big push, but only once they feel they have
>>>>> adequate resources to make the entire trip at once.
>>>>> Group B likes to frolic, and will happily take two sptens  
>>>>> backward and
>>>>> then 3 steps forward every day.
>>>>> Group C will walk forward with you at a steady pace, and  
>>>>> occasionally even
>>>>> take a step back before moving forward, but only if the path is  
>>>>> clear and
>>>>> not very steap.
>>>>> : I bet, if you did a poll of all Lucene users, you would find  
>>>>> a majority of
>>>>> : them still only run 1.4.3, or maybe 1.9. Even with 2.0, 2.3,  
>>>>> or 3.0, that is
>>>>> : still going to be the case.
>>>>> That's probably true, but a nice perk of our current backwards
>>>>> compatibility commitments is that when people pop up asking  
>>>>> questions
>>>>> about 1.4.3, we can give them like "upgrading to 2.0.0 solves your
>>>>> problem" and that advice isn't a death sentence -- the steps to  
>>>>> move
>>>>> forward are small and easy.
>>>>> I look at things the way things like Maven v1 vs v2 worked out,  
>>>>> and how
>>>>> that fractured the community for a long time (as far as i can  
>>>>> tell it's
>>>>> still pretty fractured) because the path from v1 to v2 was so  
>>>>> steep and
>>>>> involved backtracking so much and i worry that if we make  
>>>>> changes to our
>>>>> "copatibility pledge" that don't allow for an even forward  
>>>>> walk, we'll
>>>>> wind up with a heavily fractured community.
>>>>> -Hoss
>>>>> ------------------------------------------------------------------ 
>>>>> ---
>>>>> To unsubscribe, e-mail:
>>>>> For additional commands, e-mail:
>>>> ------------------------------------------------------------------- 
>>>> --
>>>> To unsubscribe, e-mail:
>>>> For additional commands, e-mail:
>>> -------------------------------------------------------------------- 
>>> -
>>> To unsubscribe, e-mail:
>>> For additional commands, e-mail:
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message