lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-7834) BlockTree's terms index should be loaded into memory lazily
Date Thu, 01 Jun 2017 10:31:04 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032759#comment-16032759
] 

Michael McCandless commented on LUCENE-7834:
--------------------------------------------

I don't think we should make lazy-loading of the terms index possible.  Lucene used to do
this and we moved away from it, long ago, for all the traps mentioned so far.

Another trap: the memory needed by your search app would become less predictable, because
you would feel successful that you were able to open the reader and run searches, and then
later on when the lazy load trips, more heap is used, possibly leading to OOME.

Loading the FST is also very low cost, a single seek and then sequential read of N bytes.

> BlockTree's terms index should be loaded into memory lazily
> -----------------------------------------------------------
>
>                 Key: LUCENE-7834
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7834
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/codecs
>            Reporter: David Smiley
>            Assignee: David Smiley
>         Attachments: LUCENE_7834_BlockTreeLazyFST.patch
>
>
> I propose that BlockTree delay loading the FST prefix terms index into memory until {{terms(String)}}
is first called.  This seems like how it should work since if someone wants to eager load
then they can use {{IndexReaderWarmer}}.  By making the FST lazy load, we can be more NRT
friendly (if some fields are rarely used), and also save memory (if some fields are rarely
used).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message