lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ankit Jain (JIRA)" <>
Subject [jira] [Created] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap
Date Fri, 11 Jan 2019 18:59:00 GMT
Ankit Jain created LUCENE-8635:

             Summary: Lazy loading Lucene FST offheap using mmap
                 Key: LUCENE-8635
             Project: Lucene - Core
          Issue Type: New Feature
          Components: core/FSTs
         Environment: I used below setup for es_rally tests:

single node i3.xlarge running ES 6.5

es_rally was running on another i3.xlarge instance
            Reporter: Ankit Jain
         Attachments: offheap.patch, rally_benchmark.xlsx

Currently, FST loads all the terms into heap memory during index open. This causes frequent
JVM OOM issues if the term size gets big. A better way of doing this will be to lazily load
FST using mmap. That ensures only the required terms get loaded into memory.

Lucene can expose API for providing list of fields to load terms offheap. I'm planning to
take following approach for this:
 # Add a boolean property fstOffHeap in FieldInfo
 # Pass list of offheap fields to lucene during index open (ALL can be special keyword for
loading ALL fields offheap)
 # Initialize the fstOffHeap property during lucene index open
 # FieldReader invokes default FST constructor or OffHeap constructor based on fstOffHeap

I created a patch (that loads all fields offheap), did some benchmarks using es_rally and
results look good.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message