lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amrit Sarkar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-11200) provide a config to enable disable ConcurrentMergeSchedule.doAutoIOThrottle
Date Mon, 07 Aug 2017 19:47:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117144#comment-16117144
] 

Amrit Sarkar commented on SOLR-11200:
-------------------------------------

The above patch works just fine, the logging isn't obvious which created significant confusion
::
{code}
2017-08-07 17:18:18.203 INFO  (Lucene Merge Thread #1) [c:collection1 s:shard1 r:core_node2
x:collection1_shard1_replica_n1] o.a.s.u.LoggingInfoStream [MergeScheduler][Lucene Merge Thread
#1]: updateMergeThreads ioThrottle=false targetMBPerSec=20.0 MB/sec
merge thread Lucene Merge Thread #0 estSize=3.2 MB (written=2.0 MB) runTime=156.3s (stopped=0.0s,
paused=0.0s) rate=unlimited
  leave running at Infinity MB/sec
{code}
Even if the targetMBPersec=20, the merges are happening at {{rate=unlimited}}, maximum possible
disk write speed.
{code}
merge thread Lucene Merge Thread #0 estSize=29.4 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #1 estSize=77.6 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #2 estSize=86.6 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #1 estSize=77.6 MB (written=76.1 MB) runTime=10.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #2 estSize=86.6 MB (written=1.0 MB) runTime=-0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #3 estSize=133.9 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #3 estSize=133.9 MB (written=132.3 MB) runTime=12.8s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #4 estSize=71.9 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #4 estSize=71.9 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #5 estSize=82.0 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #6 estSize=92.5 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #7 estSize=128.2 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #7 estSize=128.2 MB (written=117.2 MB) runTime=12.2s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #8 estSize=66.7 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #8 estSize=66.7 MB (written=21.1 MB) runTime=0.8s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #9 estSize=206.2 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #8 estSize=66.7 MB (written=65.1 MB) runTime=9.2s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #9 estSize=206.2 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #9 estSize=206.2 MB (written=191.5 MB) runTime=15.3s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #10 estSize=146.7 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #10 estSize=146.7 MB (written=47.3 MB) runTime=1.9s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #11 estSize=280.9 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #10 estSize=146.7 MB (written=143.3 MB) runTime=17.2s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #11 estSize=280.9 MB (written=1.0 MB) runTime=0.3s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #11 estSize=280.9 MB (written=143.3 MB) runTime=12.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #12 estSize=100.8 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #11 estSize=280.9 MB (written=208.3 MB) runTime=24.4s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #11 estSize=280.9 MB (written=235.3 MB) runTime=28.2s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #13 estSize=193.6 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #13 estSize=193.6 MB (written=79.3 MB) runTime=4.7s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #13 estSize=193.6 MB (written=155.4 MB) runTime=16.1s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #14 estSize=35.5 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #13 estSize=193.6 MB (written=190.4 MB) runTime=20.3s (stopped=0.0s,
paused=0.0s) rate=unlimited
merge thread Lucene Merge Thread #15 estSize=94.2 MB (written=0.0 MB) runTime=0.0s (stopped=0.0s,
paused=0.0s) rate=unlimited
{code}

I am running some tests, to benchmark and see if we this or altering factors increase the
indexing speed. I will post when I have something substantial.

> provide a config to enable disable ConcurrentMergeSchedule.doAutoIOThrottle
> ---------------------------------------------------------------------------
>
>                 Key: SOLR-11200
>                 URL: https://issues.apache.org/jira/browse/SOLR-11200
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Nawab Zada Asad iqbal
>            Priority: Minor
>         Attachments: SOLR-11200.patch, SOLR-11200.patch
>
>
> This config can be useful while bulk indexing. Lucene introduced it https://issues.apache.org/jira/browse/LUCENE-6119
. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message