lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sreedevi s <sreedevi.payik...@gmail.com>
Subject Re: MMapDirectory or FSDirectory
Date Thu, 05 Feb 2015 10:13:43 GMT
Hi,
Thank you for sharing the blog.I am using FSDirectory.open() in my
program.So, I guess I am using MMapDirectory. It takes about 3 minutes when
I search for a key(which is actually present in 80% of total data) in all
the fields(1000) in this 1 million documents.

Best Regards,
Sreedevi S

On Thu, Feb 5, 2015 at 3:20 PM, Uwe Schindler <uwe@thetaphi.de> wrote:

> Hi,
>
> If you use FSDirectory.open() it will automatically choose MMapDirectory
> on 64 bit systems. Please note, virtual memory is != physical RAM. A 64 bit
> machine has *always* >1 Terabyte of virtual address space available, this
> is unrelated to physical memory (a common misunderstanding about mmap):
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> The speed difference depends on use case: In general MMapDirectory is much
> faster in multi-threaded environments, because no concurrency problems. If
> you use SimpleFSDirectory this is the largest bootleneck. NIOFSDirectory
> does not have concurrency problems, but it is still slower because it does
> a lot of extra copying of data between kernel space and user space for
> buffering. MMapDirectory is muuuuuuuch faster if you sort by docvalues
> fields, because it supports random access without any buffering overhead.
>
> So please: Use MMapDirectory where possible - this is completely unrelated
> to how much RAM you have available!
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: sreedevi s [mailto:sreedevi.payikkad@gmail.com]
> > Sent: Thursday, February 05, 2015 10:13 AM
> > To: java-user@lucene.apache.org
> > Subject: MMapDirectory or FSDirectory
> >
> > Hi,
> > I am doing some performance analysis with lucene. I have 1 million
> resources
> > with 1000 attributes.
> > According to how I index, I will have 1 million documents with 1000
> fields.
> > For me the total data was about 100 GB and while using FSDirectory to
> store
> > my indices, index size was almost 6 GB.
> > I have virtual memory available of almost 8 GB. Is it advised to use
> > MMapDirectory for increased performance?
> > Many blogs suggest it doesnt bring out much performance difference.
> >
> >
> > Best Regards,
> > Sreedevi S
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message