lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-2898) CMS merge throttling is not aggressive enough
Date Sun, 30 Jan 2011 12:22:43 GMT


Michael McCandless commented on LUCENE-2898:

bq. So now the code will wait until one of the merges completes, before the user gets back
control (since it's CMS.merge(IW) that waits)?

Right, that's exactly what we want to happen, I think?

Ie, that thread is in CMS because it is a thread responsible somehow for adding new segments
to the index (ie, it called getReader or commit or expungeDeletes, etc.), and we are in a
situation where too many merge threads are running, so we forcefully stall the incoming thread.
 This is the last line of defense that CMS has to prevent over-production of segments (vs
rate of consumption from merging).

> CMS merge throttling is not aggressive enough
> ---------------------------------------------
>                 Key: LUCENE-2898
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1, 4.0
> I hit this crab while working on the NRT benchmarker (in luceneutil).
> CMS today forcefully idles any incoming threads, when there are too many merges pending.
> This is the last line of defense that it has, since it also juggles thread priorities
(and forcefully idles the biggest merges) to try to reduce the outstanding merge count.
> But when it cannot keep up it has no choice but to stall those threads responsible for
making new segments.
> However, the logic is in the wrong place now -- the stalling happens after pulling the
next merge from IW.  This is poor because it means if you have N indexing threads, you allow
max + N outstanding merges.
> I have a simple fix, which is to just move the stall logic to before we pull the next
merge from IW.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message