lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] [Updated] (LUCENE-5310) Merge Threads unnecessarily block on SerialMergeScheduler
Date Sat, 26 Apr 2014 22:36:14 GMT


Michael McCandless updated LUCENE-5310:

    Attachment: LUCENE-5310.patch

Another iteration ... this patch moves the "stall indexing threads if
necessary" logic to the base class (MergeScheduler) and fixes both SMS
and CMS to use it.

It also enables an app to subclass either SMS or CMS and override the
default adversary/denial-of-service protection, i.e. hard stall of
indexing threads "responsible" for making segments when too many
merges are running/scheduled.  If an app has a gentler way of reducing
the merge load (opening NRT readers less frequently, removing merge IO
rate throttling, etc.) then it can subclass and do that instead.

The behavior of CMS is unchanged, i.e. it blocks indexing threads when
the max number of merges are running and another merge wants to kick
off.  SMS is a bit improved: its merge method is no longer sync'd, and
it now only blocks if there is already a merge running and another
wants to kick off; previously it blocked if a merge was already
running regardless of whether another merge needed to run.

I think there are further simplifications we can make to CMS but we
should take those up separately.

> Merge Threads unnecessarily block on SerialMergeScheduler
> ---------------------------------------------------------
>                 Key: LUCENE-5310
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 4.5, 5.0
>            Reporter: Simon Willnauer
>            Priority: Minor
>             Fix For: 4.9, 5.0
>         Attachments: LUCENE-5310.patch, LUCENE-5310.patch, LUCENE-5310.patch, LUCENE-5310.patch,
> I have been working on a high level merge multiplexer that shares threads across different
IW instances and I came across the fact that SerialMergeScheduler actually blocks incoming
thread is a merge in going on. Yet this blocks threads unnecessarily since we pull the merges
in a loop anyway. We should use a tryLock operation instead of syncing the entire method?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message