lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravikumar Govindarajan <ravikumar.govindara...@gmail.com>
Subject Re: Can ByteBufferIndexInput use buffering?
Date Fri, 21 Oct 2016 06:11:39 GMT
Thanks Mike & Uwe for the clarification...

We have SSDs in production & currently using normal file-reads (NIOFS...)
with 64KB buffers. The time taken for disk-reads is a small part of overall
time taken (init, parsing, CPU etc...), especially for searches. A tiny
sample of the production log is here...

QueryTime:49ms. | No of Results:1 | DiskReadCount=245 | DiskReadTimeInMs=17ms

QueryTime:75ms. | No of Results:2 | DiskReadCount=234 | DiskReadTimeInMs=19ms

QueryTime:64ms. | No of Results:57 | DiskReadCount=628 | DiskReadTimeInMs=13ms

QueryTime:70ms. | No of Results:18 | DiskReadCount=190 | DiskReadTimeInMs=35ms

QueryTime:58ms. | No of Results:1 | DiskReadCount=531 | DiskReadTimeInMs=22ms


We were using fairly large buffer size, but so far the 'DiskReadTime"
seems to be in control (May be due to page cache). However, there is a
considerable read-amplification on SSDs


It was when some part of troubleshooting exercise that we thought we should
switch over to Mmap. I have a legacy API using BufferedIndexInput & was
required to wrap Mmap index-input using that & that's when I ran into this
buffering issue. Since it had a 'supposedly' benign effect, I thought may
be could buffer it

Like you said, it's better to push for changes in legacy code rather than
adopting clumsy buffering for ByteBufferIndexInput

--
Ravi

On Thu, Oct 20, 2016 at 9:27 PM, Uwe Schindler <uwe@thetaphi.de> wrote:

> Hi,
>
> adding buffering to ByteBufferIndexInput would not only be an
> anti-pattern, it would also slowdown. What is the sense of coping data from
> memory location A to memory location B before reading?
>
> I'd suggest to read this and understand what virtual memory and
> ByteBufferIndexInput does before trying to do anything like this:
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> Kind regards,
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Ravikumar Govindarajan [mailto:ravikumar.govindarajan@gmail.com]
> > Sent: Thursday, October 20, 2016 9:26 AM
> > To: java-user@lucene.apache.org
> > Subject: Can ByteBufferIndexInput use buffering?
> >
> > When we use NIOFSDirectory, lucene internally uses buffering via
> > BufferedIndexInput (1KB etc...) while reading from the file..
> >
> > However, for MmapDirectory (ByteBufferIndexInput) there is no such
> > buffering & data is read from the mapped bytes directly...
> >
> > Will it be too much of a performance drag if I wrap ByteBufferIndexInput
> > with a BufferedIndex? I mean like, is it an anti-pattern of zero-copy
> reads
> > etc...
> >
> > Any help is much appreciated
> >
> > --
> > Ravi
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message