lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Upayavira ...@odoko.co.uk>
Subject Re: SolrCloud large transaction logs
Date Thu, 10 Jan 2013 23:53:30 GMT
Thx, that makes a lot of sense.

Upayavira

On Thu, Jan 10, 2013, at 09:37 PM, Mark Miller wrote:
> I actually think the example NRT setting of one second is probably lower
> than it should be.
> 
> When you think about most NRT cases, do you really need 1 second
> visibility? You normally could easily handle 10, 20, 30 seconds or more.
> Rather than just going for the low time, I think people should consider
> the highest they can go. You will get better performance, better caching,
> and in most use cases, no worse functionality.
> 
> One second and less, IMO, should only be used when you critically require
> it for some esoteric reason. 
> 
> At least if you are designing for scale. For simple or small use cases,
> you might not worry about it.
> 
> - Mark
> 
> On Jan 10, 2013, at 2:33 PM, Upayavira <uv@odoko.co.uk> wrote:
> 
> > Heh, the "it depends" answer :-)
> > 
> > Thanks for the clarification.
> > 
> > Upayavira
> > 
> > On Thu, Jan 10, 2013, at 05:01 PM, Mark Miller wrote:
> >> I think it really depends - if you are gong for very fast visibility,
> >> your going to spend a bunch of time warming, and then just throw it out
> >> before it even gets much if any reuse. For very fast visibility
> >> turnaround, I suspect you don't want to do any warming. I think it
> >> depends on many things.
> >> 
> >> One of the tradeoffs off using a very fast soft commit is that Sol's std
> >> caches will not be nearly as useful.
> >> 
> >> - Mark
> >> 
> >> On Jan 10, 2013, at 11:24 AM, Upayavira <uv@odoko.co.uk> wrote:
> >> 
> >>> That's great Mark. Thx. One final question... all the stuff to do with
> >>> autowarming and static warming of caches - I presume all of that
> >>> configuration is still relevant (if less so) as you still need to warm
> >>> caches on a soft commit, even if those caches are much smaller than they
> >>> would be otherwise?
> >>> 
> >>> Thanks! Upayavira
> >>> 
> >>> On Thu, Jan 10, 2013, at 04:18 PM, Mark Miller wrote:
> >>>> There is no need to open a Searcher because you are controlling
> >>>> visibility through the faster 'soft' commit. That will reopen the reader
> >>>> from the IndexWriter. Because of that, there is no reason to do a heavy,
> >>>> non NRT Searcher reopen on hard commits. Essentially, the hard commit
> >>>> becomes simply about periodically flushing the tlog and the soft commit
> >>>> completely controls visibility.
> >>>> 
> >>>> - Mark
> >>>> 
> >>>> On Jan 10, 2013, at 9:41 AM, Upayavira <uv@odoko.co.uk> wrote:
> >>>> 
> >>>>> And you don't need to open a searcher (openSearcher=false) because
> >>>>> you've got caches built up already alongside the in-memory NRT segment
> >>>>> which you can continue to use once the hard commit has happened?
Is that
> >>>>> correct?
> >>>>> 
> >>>>> (sorry for hijacking the thread - hopefully it is somewhat relevant)
> >>>>> 
> >>>>> Upayavira
> >>>>> 
> >>>>> On Thu, Jan 10, 2013, at 02:18 PM, Mark Miller wrote:
> >>>>>> Setup hard auto commit with openSeacher=false. I would do it
at least
> >>>>>> once a minute. Don't worry about the commit being out of sync
on the
> >>>>>> different nodes - you will be using soft commits for visibility.
The hard
> >>>>>> commits will just be about relieving the pressure on the tlog.
> >>>>>> 
> >>>>>> - Mark
> >>>>>> 
> >>>>>> On Jan 10, 2013, at 6:43 AM, gadde <gadde.sai@gmail.com>
wrote:
> >>>>>> 
> >>>>>>> we have a SolrCloud with 3 nodes. we add documents to leader
node and use
> >>>>>>> commitwithin(100secs) option in SolrJ to add documents.
AutoSoftCommit in
> >>>>>>> SolrConfig is 1000ms.
> >>>>>>> 
> >>>>>>> Transaction logs on replicas grew bigger than the index
and we ran out of
> >>>>>>> disk space in few days. Leader's tlogs are very small in
few hundred MBs.
> >>>>>>> 
> >>>>>>> The following post suggest hard commit is required for "relieving
the memory
> >>>>>>> pressure of the transactionlog"
> >>>>>>> http://lucene.472066.n3.nabble.com/SolrCloud-is-softcommit-cluster-wide-for-the-collection-td4021584.html#a4021631
> >>>>>>> 
> >>>>>>> what is the best way to do a hard commit on this setup in
SolrCloud?
> >>>>>>> 
> >>>>>>> a. Through autoCommit in SolrConfig? which would cause hard
commit on all
> >>>>>>> the nodes at different times b. Trigger hard commit on leader
while updating
> >>>>>>> through SolrJ?
> >>>>>>> 
> >>>>>>> Thanks
> >>>>>>> Shyam
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> --
> >>>>>>> View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-large-transaction-logs-tp4032160.html
> >>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
> >>>>>> 
> >>>> 
> >> 
> 

Mime
View raw message