lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: Optimization taking days/weeks
Date Sun, 02 Mar 2008 00:15:22 GMT
It's really about the combination of index size, hardware, required response rate, query rate
and complexity.  You typically try to benchmark this stuff to see where the limit or where
the sweet spot is for your hardware.  Unfortunately, I don't have an explanation for the sudden
jump in increase time.  This might be something to take directly to java-user@lucene list,
as optimization is really pure Lucene thing.

Otis 
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----
> From: F Knudson <fknudson@lanl.gov>
> To: solr-user@lucene.apache.org
> Sent: Friday, February 29, 2008 11:38:31 AM
> Subject: Re: Optimization taking days/weeks
> 
> 
> We are a bit concerned regarding the index size.  At least no response (so
> far) as indicated that the size is unmanagable.  We killed the process -
> will move to Java6 - and will use vmstat to monitor the new optimization
> process. 
> At what index size would you begin to worry?  Or is it a combination of
> index size, optimization time, and response time?
> We are data rich here!
> Thanks
> Frances
> 
> 
> Otis Gospodnetic wrote:
> > 
> > That's a tiny little index there ;)  Circa 100GB?
> >  
> > What do you see if you run vmstat 2 while the optimization is happening?
> > Non-idle CPU?  A pile of IO?  Is there a reason for such a small heap on a
> > machine with 32GB of RAM?
> > 
> > Otis
> > 
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> > 
> > ----- Original Message ----
> >> From: F Knudson 
> >> To: solr-user@lucene.apache.org
> >> Sent: Thursday, February 28, 2008 9:54:50 AM
> >> Subject: Optimization taking days/weeks
> >> 
> >> 
> >> Optimization time on solr index has turned into days/weeks.
> >> We are using solr 1.2.
> >> We use one box to build/optimize indexes. This index is copied to another
> >> box for searching purposes.
> >> We welcome suggestions/comments, etc.  We are a bit stumped on this.
> >> Details are below.
> >> 
> >> Box details
> >> Proc: 8 Dual Core 2.6GHz
> >> Mem: 32 GB
> >> OS: Red Hat Linux Enterprise 4
> >> Kernel: 2.6.9-55.0.12.ELlargesmp
> >> 
> >> These are details from the index currently in use.  Search response time
> >> is
> >> very acceptable (searchers are very happy)
> >> Optimization time - 10433  (12/11/07)
> >> index size - 229486464
> >> # of records - 84960570 
> >> index directory
> >> flasher# ls -l
> >> total 229486464
> >> -rw-r--r--   1 flknud   staff    22197926593 Dec 12 08:07 _2bl6.fdt
> >> -rw-r--r--   1 flknud   staff    679684560 Dec 12 08:20 _2bl6.fdx
> >> -rw-r--r--   1 flknud   staff        208 Dec 12 08:23 _2bl6.fnm
> >> -rw-r--r--   1 flknud   staff    40176405625 Dec 12 09:28 _2bl6.frq
> >> -rw-r--r--   1 flknud   staff    594723994 Dec 12 09:41 _2bl6.nrm
> >> -rw-r--r--   1 flknud   staff    47616340310 Dec 12 12:07 _2bl6.prx
> >> -rw-r--r--   1 flknud   staff    76708079 Dec 12 12:25 _2bl6.tii
> >> -rw-r--r--   1 flknud   staff    6154384415 Dec 12 12:42 _2bl6.tis
> >> -rw-r--r--   1 flknud   staff         20 Dec 12 12:48 segments.gen
> >> -rw-r--r--   1 flknud   staff         44 Dec 12 12:48 segments_2c64
> >> --------------------------
> >> 
> >> current directory listing
> >> indexed new records - Jan 22 and Jan 27
> >> # of records - 85032470
> >> optimization time - 558188
> >> 
> >> There were no out of memory errors.  There was 800961792KB  left in the
> >> directory.  The files were not 
> >> collapsed as expected.  There are still files dated Jan 22 and Jan 27.  
> >> 
> >> A new optimization was started Feb. 11 and continues.
> >> This is a snapshot of the index directory.
> >> 
> >> We have at least another million records to add.  Plus weekly updates of
> >> approximately 103K records.
> >> We are using the direct indexing method.
> >> java settings used - java -Xmx1024M -Xms1024M 
> >> 
> >> The files continue to grow so work is progressing.
> >> snapshot 2/21/08
> >> -bash-3.00$ ls -ltr
> >> total 205396680
> >> -rw-r--r--  1 flknud users         208 Jan 10 07:15 _2bm7.fnm
> >> -rw-r--r--  1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt
> >> -rw-r--r--  1 flknud users   679819760 Jan 10 08:09 _2bm7.fdx
> >> -rw-r--r--  1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq
> >> -rw-r--r--  1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx
> >> -rw-r--r--  1 flknud users  6155230704 Jan 16 18:16 _2bm7.tis
> >> -rw-r--r--  1 flknud users    76704158 Jan 16 18:16 _2bm7.tii
> >> -rw-r--r--  1 flknud users   594842294 Jan 16 18:18 _2bm7.nrm
> >> -rw-r--r--  1 flknud users         208 Jan 22 08:57 _2bpa.fnm
> >> -rw-r--r--  1 flknud users    10806426 Jan 22 08:57 _2bpa.fdt
> >> -rw-r--r--  1 flknud users      371200 Jan 22 08:57 _2bpa.fdx
> >> -rw-r--r--  1 flknud users    21114330 Jan 22 08:57 _2bpa.frq
> >> -rw-r--r--  1 flknud users    25683573 Jan 22 08:57 _2bpa.prx
> >> -rw-r--r--  1 flknud users     9225592 Jan 22 08:57 _2bpa.tis
> >> -rw-r--r--  1 flknud users      118660 Jan 22 08:57 _2bpa.tii
> >> -rw-r--r--  1 flknud users      324804 Jan 22 08:57 _2bpa.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 22 09:00 _2bpl.fnm
> >> -rw-r--r--  1 flknud users     1335931 Jan 22 09:00 _2bpl.fdt
> >> -rw-r--r--  1 flknud users       36800 Jan 22 09:00 _2bpl.fdx
> >> -rw-r--r--  1 flknud users     2646708 Jan 22 09:00 _2bpl.frq
> >> -rw-r--r--  1 flknud users     3781824 Jan 22 09:00 _2bpl.prx
> >> -rw-r--r--  1 flknud users     1429176 Jan 22 09:00 _2bpl.tis
> >> -rw-r--r--  1 flknud users       18582 Jan 22 09:00 _2bpl.tii
> >> -rw-r--r--  1 flknud users       32204 Jan 22 09:00 _2bpl.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 22 09:01 _2bpm.fnm
> >> -rw-r--r--  1 flknud users      121716 Jan 22 09:01 _2bpm.fdt
> >> -rw-r--r--  1 flknud users        3200 Jan 22 09:01 _2bpm.fdx
> >> -rw-r--r--  1 flknud users      205961 Jan 22 09:01 _2bpm.frq
> >> -rw-r--r--  1 flknud users      302114 Jan 22 09:01 _2bpm.prx
> >> -rw-r--r--  1 flknud users      233641 Jan 22 09:01 _2bpm.tis
> >> -rw-r--r--  1 flknud users        3036 Jan 22 09:01 _2bpm.tii
> >> -rw-r--r--  1 flknud users        2804 Jan 22 09:01 _2bpm.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 27 14:00 _2bpn.fnm
> >> -rw-r--r--  1 flknud users      227962 Jan 27 14:00 _2bpn.fdt
> >> -rw-r--r--  1 flknud users        7200 Jan 27 14:00 _2bpn.fdx
> >> -rw-r--r--  1 flknud users      437798 Jan 27 14:00 _2bpn.frq
> >> -rw-r--r--  1 flknud users      593858 Jan 27 14:00 _2bpn.prx
> >> -rw-r--r--  1 flknud users      516031 Jan 27 14:00 _2bpn.tis
> >> -rw-r--r--  1 flknud users        6814 Jan 27 14:00 _2bpn.tii
> >> -rw-r--r--  1 flknud users        6304 Jan 27 14:00 _2bpn.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpo.fnm
> >> -rw-r--r--  1 flknud users      231456 Jan 27 14:01 _2bpo.fdt
> >> -rw-r--r--  1 flknud users        7200 Jan 27 14:01 _2bpo.fdx
> >> -rw-r--r--  1 flknud users      448401 Jan 27 14:01 _2bpo.frq
> >> -rw-r--r--  1 flknud users      616557 Jan 27 14:01 _2bpo.prx
> >> -rw-r--r--  1 flknud users      587697 Jan 27 14:01 _2bpo.tis
> >> -rw-r--r--  1 flknud users        7801 Jan 27 14:01 _2bpo.tii
> >> -rw-r--r--  1 flknud users        6304 Jan 27 14:01 _2bpo.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpp.fnm
> >> -rw-r--r--  1 flknud users      229944 Jan 27 14:01 _2bpp.fdt
> >> -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpp.fdx
> >> -rw-r--r--  1 flknud users      462865 Jan 27 14:01 _2bpp.frq
> >> -rw-r--r--  1 flknud users      652003 Jan 27 14:01 _2bpp.prx
> >> -rw-r--r--  1 flknud users      574096 Jan 27 14:01 _2bpp.tis
> >> -rw-r--r--  1 flknud users        7452 Jan 27 14:01 _2bpp.tii
> >> -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpp.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpq.fnm
> >> -rw-r--r--  1 flknud users       53438 Jan 27 14:01 _2bpq.fdt
> >> -rw-r--r--  1 flknud users        1600 Jan 27 14:01 _2bpq.fdx
> >> -rw-r--r--  1 flknud users      103059 Jan 27 14:01 _2bpq.frq
> >> -rw-r--r--  1 flknud users      157149 Jan 27 14:01 _2bpq.prx
> >> -rw-r--r--  1 flknud users      179711 Jan 27 14:01 _2bpq.tis
> >> -rw-r--r--  1 flknud users        2489 Jan 27 14:01 _2bpq.tii
> >> -rw-r--r--  1 flknud users        1404 Jan 27 14:01 _2bpq.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpr.fnm
> >> -rw-r--r--  1 flknud users      200777 Jan 27 14:01 _2bpr.fdt
> >> -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpr.fdx
> >> -rw-r--r--  1 flknud users      416658 Jan 27 14:01 _2bpr.frq
> >> -rw-r--r--  1 flknud users      570959 Jan 27 14:01 _2bpr.prx
> >> -rw-r--r--  1 flknud users      489593 Jan 27 14:01 _2bpr.tis
> >> -rw-r--r--  1 flknud users        6451 Jan 27 14:01 _2bpr.tii
> >> -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpr.nrm
> >> -rw-r--r--  1 flknud users         236 Jan 27 14:01 segments_2cap
> >> -rw-r--r--  1 flknud users          20 Jan 27 14:01 segments.gen
> >> -rw-r--r--  1 flknud users           0 Feb 11 09:56 write.lock
> >> -rw-r--r--  1 flknud users         208 Feb 11 09:56 _2bps.fnm
> >> -rw-r--r--  1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt
> >> -rw-r--r--  1 flknud users   680259760 Feb 11 12:26 _2bps.fdx
> >> -rw-r--r--  1 flknud users    24416256 Feb 21 06:42 _2bps.tii
> >> -rw-r--r--  1 flknud users  1685057536 Feb 21 06:56 _2bps.tis
> >> -rw-r--r--  1 flknud users 29885433856 Feb 21 07:32 _2bps.frq
> >> -rw-r--r--  1 flknud users 38229336064 Feb 21 07:32 _2bps.prx
> >> -- 
> >> View this message in context: 
> >> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >> 
> >> 
> > 
> > 
> > 
> > 
> 
> -- 
> View this message in context: 
> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15762378.html
> Sent from the Solr - User mailing list archive at Nabble.com.
> 
> 



Mime
View raw message