lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Laban (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (LUCENE-7671) Enhance UpgradeIndexMergePolicy with additional options
Date Fri, 03 Feb 2017 23:49:51 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852346#comment-15852346
] 

Keith Laban edited comment on LUCENE-7671 at 2/3/17 11:49 PM:
--------------------------------------------------------------

Yes, in a nutshell that is my goal. Ultimately I'd like to build a request handler in solr
that can enable a core to upgrade it's segments without first taking it offline or reconfiguring
the index writer. When not engaged it should have no effect, and when it is engaged it should
do minimal work at a time.

The additional bells and whistles of this PR are for backwards compatibility. 

Previous the behavior was:
1) Determine all old segments
2) Ask wrapping merge policy what to do with the old segments
3) Merge segments specified by wrapped merge policy
4) Merge the remaining old segments into a single new segment

Meaning, if you were to upgrade an old index using TierdMergePolicy or similar as the wrapped
MP and said {{w.forceMerge(Integer.MAX_INT)}}, the TMP would say nothing to do, but the UpgradeIndexMergePolicy
would then take it upon itself to merge everything down into a single segment.

Ideally if {{max number of segments > number of segments}} and the wrapped MP is happy,
the UIMP should not take it upon itself make any merge decisions and only upgrade segments
needing upgrade by rewritting each segment.

An additional decision to rely on cascading calls from the IW was chosen so that if this was
being used as the default MP and an upgrade was in progress, old segments could still be candidates
for merges issued during a commit.

The idea is loosely based on elasticsearch's ElasticsearchMergePolicy.

There should probably also be support for a Predicate to be passed for determining whether
the segment should be upgraded (rewritten), then this MP can be used for things such as deciding
to rewrite segments with a different codec.


was (Author: k317h):
Yes, in a nutshell that is my goal. Ultimately I'd like to build a request handler in solr
that can enable a core to upgrade it's segments without first taking it offline or reconfiguring
the index writer. When not engaged it should have no effect, and when it is engaged it should
do minimal work at a time.

The additional bells and whistles of this PR are for backwards compatibility. 

Previous the behavior was:
1) Determine all old segments
2) Ask wrapping merge policy what to do with the old segments
3) Merge segments specified by wrapped merge policy
4) Merge the remaining old segments into a single new segment

Meaning, if you were to upgrade an old index using TierdMergePolicy or similar as the wrapped
MP and said {{w.forceMerge(Integer.MAX_INT)}}, the TMP would say nothing to do, but the UpgradeIndexMergePolicy
would then take it upon itself to merge everything down into a single segment.

Ideally if {{max number of segments > number of segments}} and the wrapped MP is happy,
the UIMP should not take it upon itself make any merge decisions and only upgrade segments
needing upgrade. 

An additional decision to rely on cascading calls from the IW was chosen so that if this was
being used as the default MP and an upgrade was in progress, old segments could still be candidates
for merges issued during a commit.

The idea is loosely based on elasticsearch's ElasticsearchMergePolicy.

There should probably also be support for a Predicate to be passed for determining whether
the segment should be upgraded (rewritten), then this MP can be used for things such as deciding
to rewrite segments with a different codec.

> Enhance UpgradeIndexMergePolicy with additional options
> -------------------------------------------------------
>
>                 Key: LUCENE-7671
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7671
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Keith Laban
>
> Enhance UpgradeIndexMergePolicy to be a MergePolicy that can be used outside the scope
the IndexUpgrader.
> The enhancement aims to allow the UpgradeIndexMergePolicy to:
> 1) Delegate normal force merges to the underlying merge policy
> 2) Enable a flag that will explicitly tell UpgradeIndexMergePolicy when it should start
looking for upgrades.
> 3) Allow new segments to be considered to be merged with old segments, depending on underlying
MergePolicy.
> 4) Be configurable for backwards compatibility such that only segments needing an upgrade
would be considered when merging, no explicit upgrades.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message