lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <>
Subject Re: Distributed Indexes
Date Mon, 11 Feb 2008 13:02:17 GMT
Solr has a strategy using rsync that makes it relatively easy to copy  
an index around to other servers.  It uses rsync to just copy the  
diffs, so you could easily mirror this in your application.

There is no SQL backend for Lucene, but at 4mb you could certainly  
serialize it as a blob to a SQL db, but I don't see how that would  
make it faster.

Also, you said it takes a long time to create a 4mb index.  Does this  
mean you are doing something really, really complex during analysis?   
I guess what I am missing, and I think others have hinted at, is the  
big picture isn't quite clear in our minds, because the size of the  
index seems almost trivially small in Lucene terms, so we would think  
that a) It would be really fast to create the index (seconds, not  
minutes) and b) such a small index could be easily held entirely in  
memory and should easily handle a very, very, high query rate given  
reasonable hardware, which it sounds like you have.  The other piece  
that doesn't fit in my mind is it sounds like you have a fairly high  
update rate since you are getting writes before your 4mb index can be  
copied on a local network, right?  This implies that you must also  
have a lot of deletes otherwise your index would be growing  

Thus, more details on what you are doing, how you are creating your  
index, how your CF app talks to Lucene, etc. would be good.


On Feb 10, 2008, at 12:55 PM, Ruslan Sivak wrote:

> So nobody's run into anything like this before?  The need to share  
> the index between many copies of the app possibly running on  
> multiple servers?
> Russ
> Ruslan Sivak wrote:
>> The app does other things then search the index.  I'm basically  
>> using ColdFusion for the website and have four instances running on  
>> two servers for load balancing.  Each app does the searches, and  
>> the search times are small, the index is small, but it takes a long  
>> time to fully create the index (several minutes), and I would like  
>> the index to always be up to date (which is why i replicate the  
>> changes).
>> I basically cache the index for several minutes in a RamDirectory,  
>> which works quite well for performance.  If I could store the index  
>> in a SQL Table or something, I can have a single place where the  
>> index lives and atomic updates.
>> Is there a SQL Backend for the index, or should I just take the  
>> RamDirectory, serialize it and store it as a BLOB?
>> Russ
>> Erick Erickson wrote:
>>> With an index that small, I wonder why you bother with so many  
>>> copies?
>>> What kind of load are you hitting it with and how complex are the  
>>> queries?
>>> Because unless you have *very* high query rate, I'd look at why my  
>>> queries
>>> were
>>> taking so long before complexifying things this way.
>>> Best
>>> Erick
>>> On Feb 7, 2008 4:52 PM, Ruslan Sivak <> wrote:
>>>> My index is only 4mb.  Is there a SQL backend for Lucene?
>>>> Russ
>>>> Michael McCandless wrote:
>>>>> If you're able to tell Windows FRS which specific files to copy,  
>>>>> then
>>>>> SnapshotDeletionPolicy (in 2.3) should work for this.
>>>>> It basically protects a consistent snapshot of your index,  
>>>>> ensuring
>>>>> those files will not be deleted, while not blocking further  
>>>>> updates to
>>>>> the index.
>>>>> Mike
>>>>> Ruslan Sivak wrote:
>>>>>> I'm wondering if this is a problem that lucene users have already
>>>>>> tackled.  I have four copies of the application using a lucene
>>>>>> index.  They are located on two physical servers with two  
>>>>>> copies on
>>>>>> each server accessing two copies of the lucene index.  I use  
>>>>>> Windows
>>>>>> FRS (File Replication Service) to replicate the index between  
>>>>>> the two
>>>>>> servers.
>>>>>> Things work well most of the time, but sometimes, I believe under
>>>>>> load, the index doesn't get a chance to propagate before another
>>>>>> write takes place and it gets corrupted.
>>>>>> What would you recommend I use to keep the index in sync  
>>>>>> between the
>>>>>> four copies of the app?
>>>>>> Russ
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail:
>>>>>> For additional commands, e-mail:
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail:
>>>>> For additional commands, e-mail:
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail:
>>>> For additional commands, e-mail:
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

Grant Ingersoll

Lucene Helpful Hints:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message