lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: SolrCloud large transaction logs
Date Thu, 10 Jan 2013 21:37:59 GMT
I actually think the example NRT setting of one second is probably lower than it should be.

When you think about most NRT cases, do you really need 1 second visibility? You normally
could easily handle 10, 20, 30 seconds or more. Rather than just going for the low time, I
think people should consider the highest they can go. You will get better performance, better
caching, and in most use cases, no worse functionality.

One second and less, IMO, should only be used when you critically require it for some esoteric
reason. 

At least if you are designing for scale. For simple or small use cases, you might not worry
about it.

- Mark

On Jan 10, 2013, at 2:33 PM, Upayavira <uv@odoko.co.uk> wrote:

> Heh, the "it depends" answer :-)
> 
> Thanks for the clarification.
> 
> Upayavira
> 
> On Thu, Jan 10, 2013, at 05:01 PM, Mark Miller wrote:
>> I think it really depends - if you are gong for very fast visibility,
>> your going to spend a bunch of time warming, and then just throw it out
>> before it even gets much if any reuse. For very fast visibility
>> turnaround, I suspect you don't want to do any warming. I think it
>> depends on many things.
>> 
>> One of the tradeoffs off using a very fast soft commit is that Sol's std
>> caches will not be nearly as useful.
>> 
>> - Mark
>> 
>> On Jan 10, 2013, at 11:24 AM, Upayavira <uv@odoko.co.uk> wrote:
>> 
>>> That's great Mark. Thx. One final question... all the stuff to do with
>>> autowarming and static warming of caches - I presume all of that
>>> configuration is still relevant (if less so) as you still need to warm
>>> caches on a soft commit, even if those caches are much smaller than they
>>> would be otherwise?
>>> 
>>> Thanks! Upayavira
>>> 
>>> On Thu, Jan 10, 2013, at 04:18 PM, Mark Miller wrote:
>>>> There is no need to open a Searcher because you are controlling
>>>> visibility through the faster 'soft' commit. That will reopen the reader
>>>> from the IndexWriter. Because of that, there is no reason to do a heavy,
>>>> non NRT Searcher reopen on hard commits. Essentially, the hard commit
>>>> becomes simply about periodically flushing the tlog and the soft commit
>>>> completely controls visibility.
>>>> 
>>>> - Mark
>>>> 
>>>> On Jan 10, 2013, at 9:41 AM, Upayavira <uv@odoko.co.uk> wrote:
>>>> 
>>>>> And you don't need to open a searcher (openSearcher=false) because
>>>>> you've got caches built up already alongside the in-memory NRT segment
>>>>> which you can continue to use once the hard commit has happened? Is that
>>>>> correct?
>>>>> 
>>>>> (sorry for hijacking the thread - hopefully it is somewhat relevant)
>>>>> 
>>>>> Upayavira
>>>>> 
>>>>> On Thu, Jan 10, 2013, at 02:18 PM, Mark Miller wrote:
>>>>>> Setup hard auto commit with openSeacher=false. I would do it at least
>>>>>> once a minute. Don't worry about the commit being out of sync on
the
>>>>>> different nodes - you will be using soft commits for visibility.
The hard
>>>>>> commits will just be about relieving the pressure on the tlog.
>>>>>> 
>>>>>> - Mark
>>>>>> 
>>>>>> On Jan 10, 2013, at 6:43 AM, gadde <gadde.sai@gmail.com> wrote:
>>>>>> 
>>>>>>> we have a SolrCloud with 3 nodes. we add documents to leader
node and use
>>>>>>> commitwithin(100secs) option in SolrJ to add documents. AutoSoftCommit
in
>>>>>>> SolrConfig is 1000ms.
>>>>>>> 
>>>>>>> Transaction logs on replicas grew bigger than the index and we
ran out of
>>>>>>> disk space in few days. Leader's tlogs are very small in few
hundred MBs.
>>>>>>> 
>>>>>>> The following post suggest hard commit is required for "relieving
the memory
>>>>>>> pressure of the transactionlog"
>>>>>>> http://lucene.472066.n3.nabble.com/SolrCloud-is-softcommit-cluster-wide-for-the-collection-td4021584.html#a4021631
>>>>>>> 
>>>>>>> what is the best way to do a hard commit on this setup in SolrCloud?
>>>>>>> 
>>>>>>> a. Through autoCommit in SolrConfig? which would cause hard commit
on all
>>>>>>> the nodes at different times b. Trigger hard commit on leader
while updating
>>>>>>> through SolrJ?
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Shyam
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-large-transaction-logs-tp4032160.html
>>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>> 
>>>> 
>> 


Mime
View raw message