lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <>
Subject Re: Back Compatibility
Date Wed, 23 Jan 2008 01:27:33 GMT
I think there are a lot of applications using Lucene where "whether  
its lost a bit of data or not" is not acceptable.

However, it is probably fine for a web search, or intranet search.

As to your first point, that is why the really great open-source  
projects (eclipse, open office) have a financial backer that provides  
significant direction, and contributions.  They wouldn't waste their  
resources developing esoteric features with little appeal, and direct  
their resources to broader features that others can then develop  
finer features on top off.

I don't question the abilities of Michael whatsoever - I just wish  
they were directed at broader features. The review by the voters (and  
this list) allows development to focused.

Frequently perfectly correct patches are rejected by the voters. Why?  
Because SOMEONE needs to keep the development focused - if not there  
will be chaos.

On Jan 22, 2008, at 4:19 PM, Mark Miller wrote:

> I humbly disagree about NFS. Arguing about where free time was  
> invested, or wasted, or inefficient, in an open source project just  
> seems silly. One of the great benefits is esoteric work that would  
> normally not be allowed for. NFS is easy. A lot of Lucene users  
> don't care about Lucene. They just want something easy to setup. It  
> especially doesn't make send when talking about Michael. He seems  
> to spit out Lucene code in his sleep. I doubt NFS stuff did  
> anything but to make him more brilliant at manipulating Lucene. It  
> certainly hasn't made him any less prolific.
> I am very in favor of your talk about transactional support. Man do  
> I want Lucene to have that. But the fact that we are getting to  
> where the index cannot be corrupted is still a great step forward.  
> Knowing that my indexes will not be corrupted while running at a  
> place that needs access 24/7 is just wonderful. I can get something  
> working for them quick, whether its lost a bit of data or not. Now  
> full support to guarantee that my Lucene index is consistent with  
> my Database? Even better. I wish. But I am still very thankful for  
> the first step of a guaranteed consistent index.
> Your glass is always half full ;) I aspire to your crankiness when  
> I get older.
> - Mark
> robert engels wrote:
>> One more example on this. A lot of work was done on transaction  
>> support. I would argue that this falls way short of what is  
>> needed, since there is no XA transaction support. Since the lucene  
>> index (unless stored in an XA db) is a separate resource, it  
>> really needs XA support in order to be consistent with the other  
>> resources.
>> All of the transaction work that has been performed only  
>> guarantees that barring a physical hardware failure the lucene  
>> index can be opened and used at a known state.  This index though  
>> is probably not consistent with the other resources.
>> All that was done is that we can now guarantee that the index is  
>> consistent at SOME point in time.
>> Given the work that was done, we are probably closer to adding XA  
>> support, but I think this would be much easier if the concept of a  
>> transaction was made first class through the API (and then XA  
>> transactions need to be supported).
>> On Jan 22, 2008, at 2:49 PM, robert engels wrote:
>>> I don't think group C is interested in bug fixes. I just don't  
>>> see how Lucene is at all useful if the users are encountering any  
>>> bug - so they either don't use that feature, or they have already  
>>> developed a work-around (or they have patched the code in a way  
>>> that avoids the bug, yet is specific to their environment).
>>> For example, I think the NFS work (bugs, fixes, etc.) was quite  
>>> substantial. I think the actual number of people trying to use  
>>> NFS is probably very low - as the initial implementation had so  
>>> many problems (and IMO is not a very good solution for  
>>> distributed indexes anyway). So all the work in trying to make  
>>> NFS work "correctly" behind the scenes may have been inefficient,  
>>> since a more direct, yet major fix may have solved the problem  
>>> better (like distributed server support, not shared index access).
>>> I just think that trying to maintain API compatibility through  
>>> major releases is a bad idea. Leads to bloat, and complex code -  
>>> both internal and external.  In order to achieve great gains in  
>>> usability and/or performance in a mature product like Lucene  
>>> almost certainly requires massive changes to the processes,  
>>> algorithms and structures, and the API should change as well to  
>>> reflect this.
>>> On Jan 22, 2008, at 2:30 PM, Chris Hostetter wrote:
>>>> : If they are " no longer actively developing the portion of the  
>>>> code that's
>>>> : broken, aren't seeking the new feature, etc", and they stay  
>>>> back on old
>>>> : versions... isn't that exactly what we want? They can stay on  
>>>> the old version,
>>>> : and new application development uses the newer version.
>>>> This basically mirrors a philosophy that is rising in the Perl
>>>> community evangelized by (a really smart dude named chromatic) ...
>>>> "why are we worry about the effect of upgrades on users who  
>>>> don't upgrade?"
>>>> The problem is not all users are created equal and not all users  
>>>> upgrade
>>>> for the same reasons or at the same time...
>>>> Group A: If someone is paranoid about upgrading, and is still  
>>>> running
>>>> lucene1.4.3 because they are afraid if they upgrade their app  
>>>> will break
>>>> and they don't want to deal with it; they don't care about known  
>>>> bugs in
>>>> lucene1.4.3, as long as those bugs haven't impacted them yet --  
>>>> these
>>>> people aren't going to care wether we add a bunch of new methods to
>>>> interfaces, or remove a bunch of public methods from arbitrary  
>>>> releases,
>>>> because they are never going to see them.  They might do a total  
>>>> rewrite
>>>> of their project later, and they'll worry about it then (when  
>>>> they have
>>>> lots of time and QA resources)
>>>> Group: B: At the other extreme, are the "free-spirited"  
>>>> developers (god i
>>>> hate that that the word "agile" has been co-opted) who are  
>>>> always eager to
>>>> upgrade to get the latest bells and whistles, and don't mind making
>>>> changes to code and recompiling everytime they upgrades -- just  
>>>> as long as
>>>> there are some decent docs on what to change.
>>>> Croup: C: In the middle is a larg group of people who are  
>>>> interested in
>>>> upgrading, who want bug fixes, are willing to write new code to  
>>>> take
>>>> advantage of new features, in some cases are even willing to make
>>>> small or medium changes their code to get really good performance
>>>> improvements ... but they don't have a lot of time or energy to  
>>>> constantly
>>>> rewrite big chunks of their app.  For these people, knowing that  
>>>> they can
>>>> "drop in" the new version and it will work is a big reason why  
>>>> there are
>>>> willing to upgrade, and why they are willing to spend soem time
>>>> tweaking code to take advantage of the new features and the new
>>>> performacne enhaced APIs -- becuase they don't have to spend a  
>>>> lot of time
>>>> just to get the app working as well as it was before.
>>>> To draw an analogy...
>>>> Group A will stand in one place for a really long time no matter  
>>>> how easy
>>>> the path is.  Once in a great while they will decide to march  
>>>> forward
>>>> dozens of miles in one big push, but only once they feel they have
>>>> adequate resources to make the entire trip at once.
>>>> Group B likes to frolic, and will happily take two sptens  
>>>> backward and
>>>> then 3 steps forward every day.
>>>> Group C will walk forward with you at a steady pace, and  
>>>> occasionally even
>>>> take a step back before moving forward, but only if the path is  
>>>> clear and
>>>> not very steap.
>>>> : I bet, if you did a poll of all Lucene users, you would find a  
>>>> majority of
>>>> : them still only run 1.4.3, or maybe 1.9. Even with 2.0, 2.3,  
>>>> or 3.0, that is
>>>> : still going to be the case.
>>>> That's probably true, but a nice perk of our current backwards
>>>> compatibility commitments is that when people pop up asking  
>>>> questions
>>>> about 1.4.3, we can give them like "upgrading to 2.0.0 solves your
>>>> problem" and that advice isn't a death sentence -- the steps to  
>>>> move
>>>> forward are small and easy.
>>>> I look at things the way things like Maven v1 vs v2 worked out,  
>>>> and how
>>>> that fractured the community for a long time (as far as i can  
>>>> tell it's
>>>> still pretty fractured) because the path from v1 to v2 was so  
>>>> steep and
>>>> involved backtracking so much and i worry that if we make  
>>>> changes to our
>>>> "copatibility pledge" that don't allow for an even forward walk,  
>>>> we'll
>>>> wind up with a heavily fractured community.
>>>> -Hoss
>>>> ------------------------------------------------------------------- 
>>>> --
>>>> To unsubscribe, e-mail:
>>>> For additional commands, e-mail:
>>> -------------------------------------------------------------------- 
>>> -
>>> To unsubscribe, e-mail:
>>> For additional commands, e-mail:
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message