lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Littlestar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-5917) complex Query cause luene outMemory
Date Tue, 02 Sep 2014 03:10:21 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117852#comment-14117852
] 

Littlestar commented on LUCENE-5917:
------------------------------------

{code}
MultiTermQueryWrapperFilter.java
...........
@Override
  public DocIdSet getDocIdSet(AtomicReaderContext context, Bits acceptDocs) throws IOException
{
    final AtomicReader reader = context.reader();
    final Fields fields = reader.fields();
    if (fields == null) {
      // reader has no fields
      return null;
    }

    final Terms terms = fields.terms(query.field);
    if (terms == null) {
      // field does not exist
      return null;
    }

    final TermsEnum termsEnum = query.getTermsEnum(terms);
    assert termsEnum != null;
    if (termsEnum.next() != null) {
      // fill into a FixedBitSet
      final FixedBitSet bitSet = new FixedBitSet(context.reader().maxDoc()); //==================here......
      DocsEnum docsEnum = null;
      do {
        // System.out.println("  iter termCount=" + termCount + " term=" +
        // enumerator.term().toBytesString());
        docsEnum = termsEnum.docs(acceptDocs, docsEnum, DocsEnum.FLAG_NONE);
        int docid;
        while ((docid = docsEnum.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {
          bitSet.set(docid);
        }
      } while (termsEnum.next() != null);
      // System.out.println("  done termCount=" + termCount);

      return bitSet;
    } else {
      return null;
    }
  }
{code}


> complex Query cause luene outMemory
> -----------------------------------
>
>                 Key: LUCENE-5917
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5917
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/search
>    Affects Versions: 4.9
>         Environment: 128G RAM  + 1.5T SSD + 1.2T DATA(10*10000*10000 records).
>            Reporter: Littlestar
>            Priority: Minor
>
> RangeQuery, prefixQuery and WildcardQuery use FixedBitSet when TERM_COUNT >= 350 or
DOC_COUNT_PERCENT >=0.1.
> It use a lots of memory when maxDoc very large.
> MultiTermQueryWrapperFilter<Q extends MultiTermQuery> extends Filter
> a little threads run with query "a* OR b* OR c*.....OR z*“ will cause luene outMemory,
but there is no ways to prevent it.
> another thing, some complex query also use a lot of memory too..
> I think query implements Accountable(#ramSizeInBytes), users can throw a exception better
than OutOfMemory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message