lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renee Sun <renee_...@mcafee.com>
Subject Re: partial optimize does not reduce the segment number to maxNumSegments
Date Tue, 12 Apr 2011 17:57:18 GMT
Hi Hoss,
thanks for your response...

you are right I got a typo in my question, but I did use maxSegments, and
here is the exactly url I used:

 curl
'http://localhost:8080/solr/97/update?optimize=true&maxSegments=10&waitFlush=true'

I used jconsole and du -sk to monitor each partial optimize, and I am sure
the optimize was done and
it always reduce segment files from 130+ to 65+ when I started with
maxSegments=10; when I run
again with maxSegments=9, it reduce to somewhere in 50.

when I use maxSegments=2, it always reduce the segment to 18; and
maxSegments=1 (full optimize)
will always reduce the core to 10 segment files.

this has been repeated for about dozen times.

I think the resulting files number is depending on the size of the core. I
have a core takes 10GB disk
space, and it has 4 million documents.

It perhaps also depends on other sole/lucene configurations? let me know if
I should give you any data
with our solr config.  

Here is the actual data from the test I run lately for your reference, you
can see it definitely finished
each partial optimize and the time spent is also included (please note I am
using a core id there which
is different from yours):

/tmp # ls /xxx/solr/data/32455077/index | wc   ---------------> this is the
start point, 150 seg files
     150      150     946
/tmp # time curl
'http://localhost:8080/solr/32455077/update?optimize=true&maxSegments=10&waitFlush=true'
real    0m36.050s
user    0m0.002s
sys    0m0.003s

/tmp # ls /xxx/solr/data/32455077/index | wc    ---------> after first
partial optimize (10), reduce to 82
     82      82     746
/tmp # time curl
'http://localhost:8080/solr/32455077/update?optimize=true&maxSegments=9&waitFlush=true'
real    1m54.364s
user    0m0.003s
sys    0m0.002s

/tmp # ls /xxx/solr/data/32455077/index | wc
     74      74     674
/tmp # time curl
'http://localhost:8080/solr/32455077/update?optimize=true&maxSegments=8&waitFlush=true'
real    2m0.443s
user    0m0.002s
sys    0m0.003s

/tmp # ls /xxx/solr/data/32455077/index | wc
     66      66     602
/tmp # time curl
'http://localhost:8080/solr/32455077/update?optimize=true&maxSegments=7&waitFlush=true'
<?xml version="1.0" encoding="UTF-8"?>
real    3m22.201s
user    0m0.002s
sys    0m0ls 

/tmp # ls /xxx/solr/data/32455077/index | wc
     58      58     530
/tmp #  time curl
'http://localhost:8080/solr/32455077/update?optimize=true&maxSegments=6&w 
real    3m29.277s
user    0m0.001s
sys    0m0.004s

/tmp # ls /xxx/solr/data/32455077/index | wc
     50      50     458
/tmp # time curl
'http://localhost:8080/solr/32455077/update?optimize=true&maxSegments=5&w 
real    3m41.514s
user    0m0.003s
sys    0m0.003s

/tmp # ls /xxx/solr/data/32455077/index | wc
     42      42     386
/tmp # time curl
'http://localhost:8080/solr/32455077/update?optimize=true&maxSegments=4&w 
real    5m35.697s
user    0m0.003s
sys    0m0.004s

/tmp # ls /xxx/solr/data/32455077/index | wc
     34      34     314
/tmp # time curl
'http://localhost:8080/solr/32455077/update?optimize=true&maxSegments=3wa 
real    7m8.773s
user    0m0.003s
sys    0m0.002s

/tmp # ls /xxx/solr/data/32455077/index | wc 
     26      26     242
/tmp # time curl
'http://localhost:8080/solr/32455077/update?optimize=true&maxSegments=2&w 
real    9m18.814s
user    0m0.004s
sys    0m0.001s

/tmp # ls /xxx/solr/data/32455077/index | wc
     18      18     170
/tmp # time curl
'http://localhost:8080/solr/32455077/update?optimize=true&maxSegments=1&w
(full optimize)
real    16m6.599s
user    0m0.003s
sys    0m0.004s

Disk Space Usage:
first 3 runs took about 20% extra 
middle couple runs took about 50% extra 
last full optimize took 100% extra


--
View this message in context: http://lucene.472066.n3.nabble.com/partial-optimize-does-not-reduce-the-segment-number-to-maxNumSegments-tp2682195p2812415.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message