lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <>
Subject Re: Index updates between machines
Date Tue, 03 Apr 2007 19:41:42 GMT
How fast are your disks?  Perhaps they are having trouble keeping up with simultaneous searches
and massive file copying.

 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy --  -  Tag  -  Search  -  Share

----- Original Message ----
From: Chun Wei Ho <>
Sent: Tuesday, April 3, 2007 10:40:16 AM
Subject: Index updates between machines

We are running a search service on the internet using two machines. We
have a crawler machine which crawls the web and merges new documents
found into the Lucene index. We have a searcher machine which allows
users to perform searches on the Lucene index.

Periodically, we would copy the newest version of the index from the
crawler machine over to the searcher machine (via copy over a NFS
mount). The searcher would then detect the new version, close the old
index, open the new index and resume the search service.

As the index have been growing in size, we have been noticing that the
search response time on the searcher machine increases drastically
when an index (about 15GB) is being copied from the crawler to the
searcher. Both machines run Fedora Core 4 and are on a gbps lan.

We've tried a number of ways to reduce the impact of the copy over NFS
on searching performance, such as "nice"ing the copy process, but to
no avail. I wonder if anyone is running a lucene search service over a
similar architecture and how you are managing the updates to the
lucene index.



To unsubscribe, e-mail:
For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message