lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: unstable results on refresh
Date Wed, 22 Oct 2014 13:48:51 GMT
I would rather ask whether such small differences matter enough to
do this. Is this something users will _ever_ notice? Optimization
is quite a heavyweight operation, and is generally not recommended
on indexes that change often, and 5 minutes is certainly below
the recommendation for optimizing.

There is/has been work done on "distributed IDF", but I don't quite
know the current status that should address this (I think).

But other than in a test setup, is it worth the effort?

Best,
Erick

On Wed, Oct 22, 2014 at 3:54 AM, Giovanni Bricconi
<giovanni.bricconi@banzai.it> wrote:
> I have made some small patch to the application to make this problem less
> visible, and I'm trying to perform the optimize once per hour, yesterday it
> took 5 minutes to perform it, this morning 15 minutes. Today I will collect
> some statistics but the publication process sends documents every 5
> minutes, and I think the optimize is taking too much time.
>
> I have no default mergeFactor configured for this collection, do you think
> that setting it to a small value could improve the situation? If I have
> understood well having to merge segments will keep similar stats on all
> nodes. It's ok to have the indexing process a little bit slower.
>
>
> 2014-10-21 18:44 GMT+02:00 Erick Erickson <erickerickson@gmail.com>:
>
>> Giovanni:
>>
>> To see how this happens, consider a shard with a leader and two
>> followers. Assume your autocommit interval is 60 seconds on each.
>>
>> This interval can expire at slightly different "wall clock" times.
>> Even if the servers started perfectly in synch, they can get slightly
>> out of sync. So, you index a bunch of docs and these replicas close
>> the current segment and re-open a new segment with slightly different
>> contents.
>>
>> Now docs come in that replace older docs. The tf/idf statistics
>> _include_ deleted document data (which is purged on optimize). Given
>> that doc X an be in different segments (or, more accurately, segments
>> that get merged at different times on different machines), replica 1
>> may have slightly different stats than replica 2, thus computing
>> slightly different scores.
>>
>> Optimizing purges all data related to deleted documents, so it all
>> regularizes itself on optimize.
>>
>> Best,
>> Erick
>>
>> On Tue, Oct 21, 2014 at 11:08 AM, Giovanni Bricconi
>> <giovanni.bricconi@banzai.it> wrote:
>> > I noticed again the problem, now I was able to collect some data. in my
>> > paste http://pastebin.com/nVwf327c you can see the result of the same
>> query
>> > issued twice, the 2nd and 3rd group are swapped.
>> >
>> > I pasted also the clusterstate and the core state for each core.
>> >
>> > The logs did'n show any problem related to indexing, only some malformed
>> > query.
>> >
>> > After doing an optimize the problem disappeared.
>> >
>> > So, is the problem related to documents that where deleted from the
>> index?
>> >
>> > The optimization took 5 minutes to complete
>> >
>> > 2014-10-21 11:41 GMT+02:00 Giovanni Bricconi <
>> giovanni.bricconi@banzai.it>:
>> >
>> >> Nice!
>> >> I will monitor the index and try this if the problem comes back.
>> >> Actually the problem was due to small differences in score, so I think
>> the
>> >> problem has the same origin
>> >>
>> >> 2014-10-21 8:10 GMT+02:00 lboutros <boutrosl@gmail.com>:
>> >>
>> >>> Hi Giovanni,
>> >>>
>> >>> we had this problem as well.
>> >>> The cause was that the different nodes have slightly different idf
>> values.
>> >>>
>> >>> We solved this problem by doing an optimize operation which really
>> remove
>> >>> suppressed data.
>> >>>
>> >>> Ludovic.
>> >>>
>> >>>
>> >>>
>> >>> -----
>> >>> Jouve
>> >>> France.
>> >>> --
>> >>> View this message in context:
>> >>>
>> http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html
>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>
>> >>
>> >>
>>

Mime
View raw message