lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Lookback and/or time-aware Merge Policy?
Date Mon, 15 Jul 2013 11:28:19 GMT
Lookback is a good idea: you could at least gather statistics and
assess, later, whether good merges had been selected, and maybe play
"what if" games to explore if different merge selections would have
resulted in less copying.

A time-based MergeScheduler would make sense: e.g., it would allow
small merges to run any time, but big ones must wait until "after
hours".

Also, RateLimitedDirWrapper can be used to limit IO impact of ongoing
merges.  It's like a naive ionice, for merging.

Mike McCandless

http://blog.mikemccandless.com


On Mon, Jul 8, 2013 at 10:41 PM, Otis Gospodnetic
<otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> I was (re-re-re-re)-reading Mike's post about Lucene segment merges -
> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
>
> Mike mentioned lookhead as something that could possibly yield more
> optimal merges.
>
> But what about lookback? :)
>
> What if some sort of stats were kept about about which segments were
> picked for merges?  With some sort of stats in hand, could one look
> back and, knowing what happened after those merges, evaluate if more
> optimal merge choices could have been made and then use that "next
> time"?
>
> Also, what about time of day and query rates?  Very often search
> traffic follows the wave pattern, which could mean that more
> aggressive merging could be done during periods with lower query
> rates... or maybe during that time more segments could be allowed to
> live in the index, assuming that after allowing that for some time,
> the subsequent merge could be bigger/more thorough, so to speak.
>
> Thoughts?
>
> Otis
> --
> Solr & ElasticSearch Support -- http://sematext.com/
> Performance Monitoring -- http://sematext.com/spm
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message