lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Contentions observed in lucene execution
Date Thu, 18 Jul 2013 13:59:22 GMT
Lucene 2.4.x is quite ancient by now ...

FSDirectory.FSIndexInput is single-threaded in seeking/reading bytes,
which I think explains your 1 and 4.  Try using MMapDirectory, if you
are using a 64 bit JVM or if your index is tiny.  Newer Lucene
versions also have NIOFSDirectory, which is thread-friendly on Unix
(but not on Windows due to a JVM bug).

For 2 and 3, creating a FieldCache entry is also single threaded, but
this is a one-time event on the first search to the IndexReader
requiring that entry.  Lucene 4.x adds doc values which are much more
efficient to init at search time.

But, what changed in your app?  Perhaps there's less RAM available to
the OS for caching IO pages (this could explain 1 and 4)?

Mike McCandless

http://blog.mikemccandless.com


On Thu, Jul 18, 2013 at 6:46 AM, RameshIyerV <rameshiyerv@hotmail.com> wrote:
> Hi All,
>
> I need some help in analyzing some contentions we observe in the Lucene
> execution.
>
> We are supporting the Sterling 9.0 fulfillment application and it uses
> Lucene 2.4 for catalog search functionality.
>
> ---The Issue----
> This system is Live in production since Nov 2012 and only recently (Mid June
> 2013) our application is forming stuck threads during lucene invocations,
> this causes our application to crash.
>
> This occurs 2 - 3 times a week, on other days we see spikes of very slow
> performance on the exact places that causes stuck threads.
>
> ---The research---
> We have validated that the data or the usage has not grown between Jan 2012
> & now.
>
> We took snapshot of the code execution (through visual VM) and for slow
> running treads we validated that too much time is spent at certain spots
> (these very same spots appear in the stack trace of the stuck threads).
>
> ---Help needed---
> If you can guide me on what kind of contentions (heap, IO, Data, CPU, JVM
> params) can cause such a behavior it will really help.
>
>
> ---Lucene Invocation contentions observed---
> (We find stuck threads & slowness at the following spots, ordered in the
> order of severity [high to low])
> 1.      java.io.RandomAccessFile.readBytes(Native Method)
>         java.io.RandomAccessFile.read(RandomAccessFile.java:338)
>
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:596)
>
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)
>
> 2.      org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
>         org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167)
>
> org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:373)
>         org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:71)
>
> org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:351)
>
> 3.      org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:80)
>         org.apache.lucene.index.TermBuffer.read(TermBuffer.java:65)
>         org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:127)
>
> org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:389)
>         org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:71)
>
> org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:351)
>
> 4.     java.io.RandomAccessFile.seek(Native Method)
>
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:591)
>
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Contentions-observed-in-lucene-execution-tp4078796.html
> Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message