lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: SolrCloud large transaction logs
Date Thu, 10 Jan 2013 12:34:55 GMT
As I understand it, you can do a hard commit and Solr will start a new
transaction log and purge old ones. Note: Solr keeps enough transaction
logs around to re-send the last 100 updates, so depending on the frequency
you may see more than one tlog around. It doesn't matter whether
openSearcher=true or false.

I'd configure this in the solrconfig.xml file, with openSearcher=false (or
true for that matter). The fact that the commits happen at different times
shouldn't be noticeable to the users because of the NRT functionality.
Since you're running for days, you can give it a pretty long interval if
you want.... but do note that the interval between hard commits
(openSearcher=true or false) determines the window of documents that will
be in an unclosed segment, which will be thrown away if the system crashes.
They'll be replayed from the transaction log on restart, which is why they
are so huge....

Then, just for completeness' sake, I'd probably do a hard commit with
openSeacher=true at the very end of the indexing run...

FWIW,
Erick


On Thu, Jan 10, 2013 at 6:43 AM, gadde <gadde.sai@gmail.com> wrote:

> we have a SolrCloud with 3 nodes. we add documents to leader node and use
> commitwithin(100secs) option in SolrJ to add documents. AutoSoftCommit in
> SolrConfig is 1000ms.
>
> Transaction logs on replicas grew bigger than the index and we ran out of
> disk space in few days. Leader's tlogs are very small in few hundred MBs.
>
> The following post suggest hard commit is required for "relieving the
> memory
> pressure of the transactionlog"
>
> http://lucene.472066.n3.nabble.com/SolrCloud-is-softcommit-cluster-wide-for-the-collection-td4021584.html#a4021631
>
> what is the best way to do a hard commit on this setup in SolrCloud?
>
> a. Through autoCommit in SolrConfig? which would cause hard commit on all
> the nodes at different times b. Trigger hard commit on leader while
> updating
> through SolrJ?
>
> Thanks
> Shyam
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/SolrCloud-large-transaction-logs-tp4032160.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message