lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Elschot <>
Subject Delaying buffer allocation in BufferedIndexInput
Date Sat, 10 Sep 2005 20:34:26 GMT
On Friday 09 September 2005 00:34, Doug Cutting wrote:
> Paul Elschot wrote:
> > I suppose one of these cases are when many terms are used in a query. 
> > Would it be easily possible to make the buffer size for a term iterator
> > depend on the numbers of documents to be iterated?
> > Many terms only occur in a few documents, so this could be a 
> > nice win on total buffer size for the many terms case.
> This would not be too difficult.
> Look in  The buffer may be allocated when the 
> parent's stream is first cloned, but clone() won't allocate a buffer if 
> the source hasn't had a buffer allocated yet, and nothing should perform 
> i/o directly on the parent's freqStream, so in practice a buffer should 
> not be allocated until the first read is performed on the clone.

I tried delaying the buffer allocation in BufferedIndexInput by
using this clone() method:

  public Object clone() {
    BufferedIndexInput clone = (BufferedIndexInput)super.clone();
    clone.buffer = null;
    clone.bufferLength = 0;
    clone.bufferPosition = 0;
    clone.bufferStart = getFilePointer(); 
    return clone;

With this all term document iterators seem to be empty, no
query in the test cases gives any results, for example TestDemo
and TestBoolean2.
As far as I can see, this delaying should work, but it doesn't and
I have no idea why.

> So one could add an BufferedIndexInput.setBufferSize() method and then 
> call it in, when the df is known and a 
> buffer has not yet been allocated.

Indeed that looks easy enough. Now, if I could only delay
the buffer allocation...

I noticed that RAMIndexInput extends BufferedIndexInput.
It has all data in buffers already, so why is there another
layer of buffering?

Paul Elschot

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message