lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: How to make soft commit more reliable?
Date Wed, 24 Jul 2013 15:36:15 GMT
Soft commits and NRT don't exist in Solr 3.x so
I'm really confused. In 3.6 you have to do hard commits
and they're expensive 1/sec is far too often...

Erick

On Wed, Jul 24, 2013 at 5:10 AM, Mikhail Khludnev
<mkhludnev@griddynamics.com> wrote:
> Hello,
>
> First of all, I don't think it can commit (even soft) every second, afaik
> it's too frequent for typical deployment. Hence, if you really need such
> (N)RT I suggest you experiment with it right now, to face the bummer
> sooner.
> Also, one second durability sounds like over-expectation for Solr, it
> sounds like OLTP requirements.
> Then, now Solr has some sort of pre-indexing record storage called
> UpdateLog. try to experiment with syncLevel = FSYNC vs FLUSH .
> That's how it works, when document arrives for indexing it's written into
> update log, which is plain binary file. Indexing works as-is relying on
> RAMbuffer. When node dies, RAMbuffer dies, but updateLog is persistent,
> during startup Solr recovers uncommitted updates from updateLog. Caveat!
> UpdateLog has HashMap internally which easily hits OOM on rare commits.
>
>
>
> On Wed, Jul 24, 2013 at 2:56 AM, SolrLover <bbarani@gmail.com> wrote:
>
>> Currently I am using SOLR 3.5.X and I push updates to SOLR via queue
>> (Active
>> MQ) and perform hard commit every 30 minutes (since my index is relatively
>> big around 30 million documents). I am thinking of using soft commit to
>> implement NRT search but I am worried about the reliability.
>>
>> For ex: If I have the hard autocommit set to 10 minutes and a softcommit
>> every second, new documents will show up every second but in case of JVM
>> crash or power goes out I will lose all the documents after the last hard
>> commit.
>>
>> I was thinking of using a backup database or another SOLR index that I can
>> use as a backup and write the document from queue in both places (one with
>> soft commit, another index with just the push updates with normal hard
>> commits (or) write simultaneously to a db and delete the rows once the hard
>> commit is successful after making sure that we didn't lose any records).
>>
>> Does someone have any other idea to improve the reliability of the push
>> updates when using soft commit?
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/How-to-make-soft-commit-more-reliable-tp4079892.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
>  <mkhludnev@griddynamics.com>

Mime
View raw message