lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fuad Efendi" <f...@efendi.ca>
Subject FW: NativeFSLockFactory, ConcurrentMergeScheduler: why locks?
Date Tue, 11 Aug 2009 17:47:13 GMT
Most probably I need to play around UpdateHandler(s); I am using
DirectUpdateHandler with allowDuplicates = false: 
solrj.SolrServer.add(docs, overwrite=true)


Use case: I have a timestamp on a document; documents in an index get
expired by timestamp; same document could be added to the index many times
with different timestamp.
Most probably heavy 'deletes' cause this problem; will try to implement
specific UpdateHandler (can we update just a non-tokenized timestamp or
other modified field of an existing document instead of delete/insert?)


But "merge" shouldn't lock isn't it? Why new docs can't be added?

Thanks


-----Original Message-----
From: Fuad Efendi 
Sent: August-11-09 1:09 PM
To: solr-user@lucene.apache.org
Subject: NativeFSLockFactory, ConcurrentMergeScheduler: why locks?

1.       I always have files lucene-xxxx-write.lock and
lucene-xxxx-n-write.lock which I believe shouldn't be used with
NativeFSLockFactory

2.       I use mergeFactor=100 and ramBufferSizeMB=256, few GB indes size. I
tried mergeFactor=10 and mergeFactor=1000.

 

 

It seems ConcurrentMergeScheduler locks everything instead of using separate
thread on background...

 

 

So that my configured system spents half an hour to UPDATE (probably
existing in the index) million of documents, then it stops and waits few
hours for index merge which is extremely slow (a lot of deletes?)

 

With mergeFactor=1000 I had extremely performant index updates (50,000,000 a
first day), and then I was waiting more than 2 days when merge complete (and
was forced to kill process).

 

Why it locks everything?

 

Thanks,

Fuad  

 




Mime
View raw message