lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giovanni Bricconi <giovanni.bricc...@banzai.it>
Subject Re: unstable results on refresh
Date Wed, 22 Oct 2014 07:54:55 GMT
I have made some small patch to the application to make this problem less
visible, and I'm trying to perform the optimize once per hour, yesterday it
took 5 minutes to perform it, this morning 15 minutes. Today I will collect
some statistics but the publication process sends documents every 5
minutes, and I think the optimize is taking too much time.

I have no default mergeFactor configured for this collection, do you think
that setting it to a small value could improve the situation? If I have
understood well having to merge segments will keep similar stats on all
nodes. It's ok to have the indexing process a little bit slower.


2014-10-21 18:44 GMT+02:00 Erick Erickson <erickerickson@gmail.com>:

> Giovanni:
>
> To see how this happens, consider a shard with a leader and two
> followers. Assume your autocommit interval is 60 seconds on each.
>
> This interval can expire at slightly different "wall clock" times.
> Even if the servers started perfectly in synch, they can get slightly
> out of sync. So, you index a bunch of docs and these replicas close
> the current segment and re-open a new segment with slightly different
> contents.
>
> Now docs come in that replace older docs. The tf/idf statistics
> _include_ deleted document data (which is purged on optimize). Given
> that doc X an be in different segments (or, more accurately, segments
> that get merged at different times on different machines), replica 1
> may have slightly different stats than replica 2, thus computing
> slightly different scores.
>
> Optimizing purges all data related to deleted documents, so it all
> regularizes itself on optimize.
>
> Best,
> Erick
>
> On Tue, Oct 21, 2014 at 11:08 AM, Giovanni Bricconi
> <giovanni.bricconi@banzai.it> wrote:
> > I noticed again the problem, now I was able to collect some data. in my
> > paste http://pastebin.com/nVwf327c you can see the result of the same
> query
> > issued twice, the 2nd and 3rd group are swapped.
> >
> > I pasted also the clusterstate and the core state for each core.
> >
> > The logs did'n show any problem related to indexing, only some malformed
> > query.
> >
> > After doing an optimize the problem disappeared.
> >
> > So, is the problem related to documents that where deleted from the
> index?
> >
> > The optimization took 5 minutes to complete
> >
> > 2014-10-21 11:41 GMT+02:00 Giovanni Bricconi <
> giovanni.bricconi@banzai.it>:
> >
> >> Nice!
> >> I will monitor the index and try this if the problem comes back.
> >> Actually the problem was due to small differences in score, so I think
> the
> >> problem has the same origin
> >>
> >> 2014-10-21 8:10 GMT+02:00 lboutros <boutrosl@gmail.com>:
> >>
> >>> Hi Giovanni,
> >>>
> >>> we had this problem as well.
> >>> The cause was that the different nodes have slightly different idf
> values.
> >>>
> >>> We solved this problem by doing an optimize operation which really
> remove
> >>> suppressed data.
> >>>
> >>> Ludovic.
> >>>
> >>>
> >>>
> >>> -----
> >>> Jouve
> >>> France.
> >>> --
> >>> View this message in context:
> >>>
> http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html
> >>> Sent from the Solr - User mailing list archive at Nabble.com.
> >>>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message