lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-2898) CMS merge throttling is not aggressive enough
Date Sun, 30 Jan 2011 17:27:43 GMT


Michael McCandless commented on LUCENE-2898:

bq. If however MP returns N + 1 merges, then CMS will register N + 1 merges, and then stall?

Right, now I allow N+1 merges to be running, and then all incoming threads will stall.

We could change that (remove the +1) but I didn't like that because then we stall app threads
when exactly the max allowed number of merges are running.

bq. Before your changes, at most maxMergeCount merges could be registered

No, before my change, maxMergeCount *plus* the number of app threads you send into IW, could
be registered.  (This was the bug -- that our "last line of defense" was vulnerable to how
many app threads you send in).

bq. And my question I think is stil valid - what if CMS waits for nothing - meaning after
wait() finishes, IW.getNextMerge() returns null? The user thread will be blocked until then,
even though no more merges are 'pending'?

That's exactly the case I'd expect to happen.  It's not "for nothing" -- it's very much to
stall the thread that's responsible for causing new segments to be added.  The fact that after
the stall there are likely no new merges to do is fine... the app thread runs free and then
produces more segments again.

Really, if we could somehow cutover to a blocking queue... where MergePolicy is the producer
and MergeScheduler consumes, that'd be cleaner.

> CMS merge throttling is not aggressive enough
> ---------------------------------------------
>                 Key: LUCENE-2898
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.1, 4.0
> I hit this crab while working on the NRT benchmarker (in luceneutil).
> CMS today forcefully idles any incoming threads, when there are too many merges pending.
> This is the last line of defense that it has, since it also juggles thread priorities
(and forcefully idles the biggest merges) to try to reduce the outstanding merge count.
> But when it cannot keep up it has no choice but to stall those threads responsible for
making new segments.
> However, the logic is in the wrong place now -- the stalling happens after pulling the
next merge from IW.  This is poor because it means if you have N indexing threads, you allow
max + N outstanding merges.
> I have a simple fix, which is to just move the stall logic to before we pull the next
merge from IW.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message