lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jerome L Quinn <jlqu...@us.ibm.com>
Subject [1.3] help with update timeout issue?
Date Fri, 15 Jan 2010 02:59:12 GMT


Hi, folks,

I am using Solr 1.3 pretty successfully, but am running into an issue that
hits once in a long while.  I'm still using 1.3 since I have some custom
code I will have to port forward to 1.4.

My basic setup is that I have data sources continually pushing data into
Solr, around 20K adds per day.  The index is currently around 100G, stored
on local disk on a fast linux server.  I'm trying to make new docs
searchable as quickly as possible, so I currently have autocommit set to
15s.  I originally had 3s but that seems to be a little too unstable.  I
never optimize the index since optimize will lock things up solid for 2
hours, dropping docs until the optimize completes.  I'm using the default
segment merging settings.

Every once in a while I'm getting a socket timeout when trying to add a
document.  I traced it to a 20s timeout and then found the corresponding
point in the Solr log.

Jan 13, 2010 2:59:15 PM org.apache.solr.core.SolrCore execute
INFO: [tales] webapp=/solr path=/update params={} status=0 QTime=2
Jan 13, 2010 2:59:15 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
Jan 13, 2010 2:59:56 PM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@26e926e9 main
Jan 13, 2010 2:59:56 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush

Solr locked up for 41 seconds here while doing some of the commit work.
So, I have a few questions.

Is this related to GC?
Does Solr always lock up when merging segments and I just have to live with
losing the doc I want to add?
Is there a timeout that would guarantee me a write success?
Should I just retry in this situation? If so, how do I distinguish between
this and Solr just being down?
I already have had issues in the past with too many files open, so
increasing the merge factor isn't an option.


On a related note, I had previously asked about optimizing and was told
that segment merging would take care of cleaning up deleted docs.  However,
I have the following stats for my index:

numDocs : 2791091
maxDoc : 4811416

My understanding is that numDocs is the docs being searched and maxDoc is
the number of docs including ones that will disappear after optimization.
How do I get this cleanup without using optimize, since it locks up Solr
for multiple hours.  I'm deleting old docs daily as well.

Thanks for all the help,
Jerry
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message