lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ankit Jain (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (LUCENE-8783) Support FST lazy loading for non-default Codecs
Date Tue, 30 Apr 2019 00:23:00 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ankit Jain updated LUCENE-8783:
-------------------------------
    Description: Even though, [LUCENE-8635](https://issues.apache.org/jira/browse/LUCENE-8635)
and [LUCENE-8671](https://issues.apache.org/jira/browse/LUCENE-8671) adds sup  (was: Currently,
FST loads all the terms into heap memory during index open. This causes frequent JVM OOM issues
if the term size gets big. A better way of doing this will be to lazily load FST using mmap.
That ensures only the required terms get loaded into memory.

 
Lucene can expose API for providing list of fields to load terms offheap. I'm planning to
take following approach for this:
 # Add a boolean property fstOffHeap in FieldInfo
 # Pass list of offheap fields to lucene during index open (ALL can be special keyword for
loading ALL fields offheap)
 # Initialize the fstOffHeap property during lucene index open
 # FieldReader invokes default FST constructor or OffHeap constructor based on fstOffHeap
field

 
I created a patch (that loads all fields offheap), did some benchmarks using es_rally and
results look good.)

> Support FST lazy loading for non-default Codecs
> -----------------------------------------------
>
>                 Key: LUCENE-8783
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8783
>             Project: Lucene - Core
>          Issue Type: New Feature
>          Components: core/FSTs
>            Reporter: Ankit Jain
>            Priority: Major
>             Fix For: 8.0, 8.x, master (9.0)
>
>
> Even though, [LUCENE-8635](https://issues.apache.org/jira/browse/LUCENE-8635) and [LUCENE-8671](https://issues.apache.org/jira/browse/LUCENE-8671)
adds sup



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message