lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Upayavira ...@odoko.co.uk>
Subject Re: SolrCloud large transaction logs
Date Thu, 10 Jan 2013 16:24:01 GMT
That's great Mark. Thx. One final question... all the stuff to do with
autowarming and static warming of caches - I presume all of that
configuration is still relevant (if less so) as you still need to warm
caches on a soft commit, even if those caches are much smaller than they
would be otherwise?

Thanks! Upayavira

On Thu, Jan 10, 2013, at 04:18 PM, Mark Miller wrote:
> There is no need to open a Searcher because you are controlling
> visibility through the faster 'soft' commit. That will reopen the reader
> from the IndexWriter. Because of that, there is no reason to do a heavy,
> non NRT Searcher reopen on hard commits. Essentially, the hard commit
> becomes simply about periodically flushing the tlog and the soft commit
> completely controls visibility.
> 
> - Mark
> 
> On Jan 10, 2013, at 9:41 AM, Upayavira <uv@odoko.co.uk> wrote:
> 
> > And you don't need to open a searcher (openSearcher=false) because
> > you've got caches built up already alongside the in-memory NRT segment
> > which you can continue to use once the hard commit has happened? Is that
> > correct?
> > 
> > (sorry for hijacking the thread - hopefully it is somewhat relevant)
> > 
> > Upayavira
> > 
> > On Thu, Jan 10, 2013, at 02:18 PM, Mark Miller wrote:
> >> Setup hard auto commit with openSeacher=false. I would do it at least
> >> once a minute. Don't worry about the commit being out of sync on the
> >> different nodes - you will be using soft commits for visibility. The hard
> >> commits will just be about relieving the pressure on the tlog.
> >> 
> >> - Mark
> >> 
> >> On Jan 10, 2013, at 6:43 AM, gadde <gadde.sai@gmail.com> wrote:
> >> 
> >>> we have a SolrCloud with 3 nodes. we add documents to leader node and use
> >>> commitwithin(100secs) option in SolrJ to add documents. AutoSoftCommit in
> >>> SolrConfig is 1000ms.
> >>> 
> >>> Transaction logs on replicas grew bigger than the index and we ran out of
> >>> disk space in few days. Leader's tlogs are very small in few hundred MBs.
> >>> 
> >>> The following post suggest hard commit is required for "relieving the memory
> >>> pressure of the transactionlog"
> >>> http://lucene.472066.n3.nabble.com/SolrCloud-is-softcommit-cluster-wide-for-the-collection-td4021584.html#a4021631
> >>> 
> >>> what is the best way to do a hard commit on this setup in SolrCloud?
> >>> 
> >>> a. Through autoCommit in SolrConfig? which would cause hard commit on all
> >>> the nodes at different times b. Trigger hard commit on leader while updating
> >>> through SolrJ?
> >>> 
> >>> Thanks
> >>> Shyam
> >>> 
> >>> 
> >>> 
> >>> --
> >>> View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-large-transaction-logs-tp4032160.html
> >>> Sent from the Solr - User mailing list archive at Nabble.com.
> >> 
> 

Mime
View raw message