lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: unstable results on refresh
Date Tue, 21 Oct 2014 16:44:29 GMT
Giovanni:

To see how this happens, consider a shard with a leader and two
followers. Assume your autocommit interval is 60 seconds on each.

This interval can expire at slightly different "wall clock" times.
Even if the servers started perfectly in synch, they can get slightly
out of sync. So, you index a bunch of docs and these replicas close
the current segment and re-open a new segment with slightly different
contents.

Now docs come in that replace older docs. The tf/idf statistics
_include_ deleted document data (which is purged on optimize). Given
that doc X an be in different segments (or, more accurately, segments
that get merged at different times on different machines), replica 1
may have slightly different stats than replica 2, thus computing
slightly different scores.

Optimizing purges all data related to deleted documents, so it all
regularizes itself on optimize.

Best,
Erick

On Tue, Oct 21, 2014 at 11:08 AM, Giovanni Bricconi
<giovanni.bricconi@banzai.it> wrote:
> I noticed again the problem, now I was able to collect some data. in my
> paste http://pastebin.com/nVwf327c you can see the result of the same query
> issued twice, the 2nd and 3rd group are swapped.
>
> I pasted also the clusterstate and the core state for each core.
>
> The logs did'n show any problem related to indexing, only some malformed
> query.
>
> After doing an optimize the problem disappeared.
>
> So, is the problem related to documents that where deleted from the index?
>
> The optimization took 5 minutes to complete
>
> 2014-10-21 11:41 GMT+02:00 Giovanni Bricconi <giovanni.bricconi@banzai.it>:
>
>> Nice!
>> I will monitor the index and try this if the problem comes back.
>> Actually the problem was due to small differences in score, so I think the
>> problem has the same origin
>>
>> 2014-10-21 8:10 GMT+02:00 lboutros <boutrosl@gmail.com>:
>>
>>> Hi Giovanni,
>>>
>>> we had this problem as well.
>>> The cause was that the different nodes have slightly different idf values.
>>>
>>> We solved this problem by doing an optimize operation which really remove
>>> suppressed data.
>>>
>>> Ludovic.
>>>
>>>
>>>
>>> -----
>>> Jouve
>>> France.
>>> --
>>> View this message in context:
>>> http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>

Mime
View raw message